Register

Is it necessary to use Robots.txt file?

Link Building Tips & Strategies
Posts: 35
Joined: Thu Nov 15, 2018 10:59 am

Re: Is it necessary to use Robots.txt file?

Postby SearchEngineLabs » Wed Dec 19, 2018 12:22 pm

It is not necessary to have Robo.txt file but it is advisable to have Robo.txt file because it will help the google bots to crawl easily and it is an added advantage for SEO to rank high.

Posts: 57
Joined: Fri Nov 16, 2018 9:23 am

Re: Is it necessary to use Robots.txt file?

Postby jonathan brown » Mon Dec 31, 2018 7:05 am

Hi,

No, it's not necessary to add Robot.txt file but it can’t be ignored because it helps to navigate search engine robots like Google robots to crawl and index websites’ pages.

Benefits of Robot.txt file are as follows:
Index only those areas of the website you want to index.
Helps in indexing particular files on a site (images, content, videos, PDFs)
Avoids duplicate site content pages from showcasing in SERPs

Posts: 5
Joined: Thu Dec 27, 2018 6:57 am
Location: Pune

Re: Is it necessary to use Robots.txt file?

Postby lokesh91 » Wed Jan 02, 2019 5:01 am

Yes It is necessary, A robots.txt file gives instructions to web robots about the pages the website owner doesn’t wish to be crawled. This file is placed in root folder of website.

Posts: 10
Joined: Thu Dec 20, 2018 12:18 pm

Re: Is it necessary to use Robots.txt file?

Postby Shoppingswag » Wed Jan 02, 2019 9:36 am

The robots.txt protocol is a useful tool to prevent certain bots from spidering certain pages/sections of sites while allowing others through. It's also a lot easier to modify a robots.txt file than it is to update dozens, hundreds or even thousands of Web pages to make a single change. Also, unlike the META tag, the robots.txt file can block SPECIFIC search engine spiders from crawling and indexing your site.

Posts: 20
Joined: Sun Jan 06, 2019 3:42 pm
Location: Bangalore

Re: Is it necessary to use Robots.txt file?

Postby madaboutdm1719 » Sun Jan 06, 2019 3:55 pm

It is very important to use robots.txt if you are using WordPress to design your website. Simply because there might be many unwanted links, folders which you dont want google to index

Posts: 67
Joined: Fri Nov 23, 2018 8:27 am

Re: Is it necessary to use Robots.txt file?

Postby Saravanan28 » Tue Jan 08, 2019 11:15 am

yes, Robots.txt file is very important. Using a robots.txt file and with a disallow direction, we can restrict bots or search engine crawling program from websites and or from certain folders and files.

Previous

Return to Link Building

Who is online

Users browsing this forum: No registered users and 1 guest