What is robots.txt?

Basic discussions about SEO
User avatar
Posts: 21
Joined: Tue May 23, 2017 8:35 am
Location: Delhi

What is robots.txt?

Postby WebExpert » Wed Feb 28, 2018 5:49 am

hello guys, if anyone has knowledge about the robots.txt, Please share, because I don't know anything about it.

User avatar
Posts: 6
Joined: Wed Feb 28, 2018 10:01 am
Location: Hyderabad, India

Re: What is robots.txt?

Postby komali » Wed Feb 28, 2018 10:09 am

Robots.txt file is used to give instructions to bots index or not. for this, you need to submit in that file neither index or not for particular search engine bots.
Ex: User-agent: * Disallow:
User-agent: * Allow: /

Posts: 176
Joined: Wed Sep 18, 2013 3:39 am

Re: What is robots.txt?

Postby limbo » Thu Mar 01, 2018 5:39 am

If you have a proper sitemap then its irrelevant, search engine bots will crawl your site anyway unless you want to block some of your urls.

Posts: 26
Joined: Thu Jul 21, 2016 1:26 pm

Re: What is robots.txt?

Postby jackluter » Fri Mar 02, 2018 6:31 am

Robot.txt is a file which helps to secure your website payment page and security information on your website then you should submit to CMS after that crawlers and robots they did not crewel your secure information

Posts: 23
Joined: Fri Dec 01, 2017 10:15 am
Location: Mumbai

Re: What is robots.txt?

Postby rachana » Thu Mar 08, 2018 4:30 am

Robots.txt is a text file created by webmasters that instruct web robots how to crawl pages in their website. This file also indicates whether the pages of their website should be crawled or not. The crawl instructions are “allowing” or “disallowing” which are specified by user agents.

Posts: 56
Joined: Thu Jun 22, 2017 5:31 am
Location: united states

Re: What is robots.txt?

Postby andreausa » Mon Mar 12, 2018 6:59 am

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

Return to SEO General

Who is online

Users browsing this forum: No registered users and 5 guests