Register

What is robots.txt?

Basic discussions about SEO
User avatar
Posts: 21
Joined: Tue May 23, 2017 8:35 am
Location: Delhi

What is robots.txt?

Postby WebExpert » Wed Feb 28, 2018 5:49 am

hello guys, if anyone has knowledge about the robots.txt, Please share, because I don't know anything about it.

User avatar
Posts: 6
Joined: Wed Feb 28, 2018 10:01 am
Location: Hyderabad, India

Re: What is robots.txt?

Postby komali » Wed Feb 28, 2018 10:09 am

Robots.txt file is used to give instructions to bots index or not. for this, you need to submit in that file neither index or not for particular search engine bots.
Ex: User-agent: * Disallow:
User-agent: * Allow: /

Posts: 198
Joined: Wed Sep 18, 2013 3:39 am

Re: What is robots.txt?

Postby limbo » Thu Mar 01, 2018 5:39 am

If you have a proper sitemap then its irrelevant, search engine bots will crawl your site anyway unless you want to block some of your urls.

Posts: 26
Joined: Thu Jul 21, 2016 1:26 pm

Re: What is robots.txt?

Postby jackluter » Fri Mar 02, 2018 6:31 am

Robot.txt is a file which helps to secure your website payment page and security information on your website then you should submit to CMS after that crawlers and robots they did not crewel your secure information

Posts: 38
Joined: Fri Dec 01, 2017 10:15 am
Location: UAE

Re: What is robots.txt?

Postby rachana » Thu Mar 08, 2018 4:30 am

Robots.txt is a text file created by webmasters that instruct web robots how to crawl pages in their website. This file also indicates whether the pages of their website should be crawled or not. The crawl instructions are “allowing” or “disallowing” which are specified by user agents.

Posts: 60
Joined: Thu Jun 22, 2017 5:31 am
Location: united states

Re: What is robots.txt?

Postby andreausa » Mon Mar 12, 2018 6:59 am

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

Posts: 23
Joined: Mon Feb 05, 2018 9:44 am

Re: What is robots.txt?

Postby DivyaShravani » Thu Mar 29, 2018 7:06 am

Robots.txt is a text file webmasters create or passing instructions to web robots how to crawl pages on their website.
Example:
User-agent: * Disallow:
User-agent: * Allow: /

Posts: 16
Joined: Fri Feb 23, 2018 10:04 am

Re: What is robots.txt?

Postby blackmagicbaba » Thu Mar 29, 2018 9:26 am

Agree with you guys.

Posts: 135
Joined: Wed Feb 07, 2018 12:23 pm

Re: What is robots.txt?

Postby anneroberts » Tue Apr 03, 2018 11:16 am

Robot.txt file contains standard way to inform the web robot about which areas of the website should not be processed or scanned.

User avatar
Posts: 513
Joined: Mon Jan 18, 2016 5:57 pm
Location: Forum

Re: What is robots.txt?

Postby RH-Calvin » Mon Apr 09, 2018 11:25 am

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
Cheapest VPS | $1 VPS Hosting
Cheap Dedicated Hosting | Free Setup and IPMI | Unmetered Bandwidth

Next

Return to SEO General

Who is online

Users browsing this forum: No registered users and 1 guest