Register

Is it necessary to use Robots.txt file?

Link Building Tips & Strategies
Posts: 8
Joined: Sat May 13, 2017 6:28 am
Location: United States

Is it necessary to use Robots.txt file?

Postby mcafeeactivate » Thu Feb 15, 2018 1:52 pm

Hello friends- Is it necessary to use Robots.txt file? It can be affect on website ranking in search engine.

Posts: 613
Joined: Fri Jan 10, 2014 6:21 am

Re: Is it necessary to use Robots.txt file?

Postby avinash » Fri Feb 16, 2018 4:48 am

Its not utmostly necessary. If your site has a clear navigation then Google and other search engine bots will crawl your pages anyway.

Posts: 67
Joined: Fri Oct 13, 2017 12:34 pm

Re: Is it necessary to use Robots.txt file?

Postby edefiners » Wed Feb 21, 2018 11:10 am

It is not necessary but it is advisable to have a Robot.txt file for your website

Posts: 228
Joined: Tue Nov 08, 2016 5:12 am

Re: Is it necessary to use Robots.txt file?

Postby Davidwatson » Fri Mar 23, 2018 10:46 am

Reasons you may want to have a robots.txt file:

You have content you want blocked from search engines
You are using paid links or advertisements that need special instructions for robots
You want to fine tune access to your site from reputable robots
You are developing a site that is live, but you do not want search engines to index it yet
They help you follow some Google guidelines in some certain situations
You need some or all of the above, but do not have full access to your webserver and how it is configured

Posts: 27
Joined: Thu Jan 11, 2018 7:23 am
Location: Chennai, India

Re: Is it necessary to use Robots.txt file?

Postby hmsnaveen95 » Mon Mar 26, 2018 8:06 am

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically Google, bing or yahoo search engine robots) how to crawl and index pages on their website.

Posts: 658
Joined: Tue Sep 08, 2015 1:48 am
Location: Philippines

Re: Is it necessary to use Robots.txt file?

Postby steveposter » Tue Mar 27, 2018 2:43 am

I think its necessary to use Robots.txt file for those module that is not needed to be index and this is to have a smooth crawling index process within the website.

Posts: 613
Joined: Fri Jan 10, 2014 6:21 am

Re: Is it necessary to use Robots.txt file?

Postby avinash » Mon Apr 02, 2018 6:28 am

steveposter wrote:I think its necessary to use Robots.txt file for those module that is not needed to be index and this is to have a smooth crawling index process within the website.


If navigation is good the search bots will crawl and index anyway.

Posts: 135
Joined: Wed Feb 07, 2018 12:23 pm

Re: Is it necessary to use Robots.txt file?

Postby anneroberts » Mon Apr 09, 2018 8:55 am

It is not necessary to use Robots.txt file, But sometime we need to tell crawler which are the pages needs to be indexed and which is not, In that case It is necessary to use Robots.txt file to guide crawler.

User avatar
Posts: 699
Joined: Mon Jan 18, 2016 5:57 pm
Location: Forum

Re: Is it necessary to use Robots.txt file?

Postby RH-Calvin » Mon Apr 09, 2018 11:17 am

Yes robots.txt file is necessary for providing information to search engines. Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
Cheapest VPS | $1 VPS Hosting
Cheap Dedicated Servers | Free Setup and IPMI | Unmetered Bandwidth

Posts: 221
Joined: Sat Jun 30, 2018 9:10 am

Re: Is it necessary to use Robots.txt file?

Postby davidsmith21 » Tue Sep 18, 2018 5:47 am

It is not necessary to use Robot.txt file in website because it's use only when you require to disallow content or other things.

Return to Link Building

Who is online

Users browsing this forum: davidsmith21, richardmsmith and 1 guest