Register

What is the purpose of robots.txt file?

Basic discussions about SEO
Posts: 511
Joined: Sat Nov 09, 2013 2:26 pm
Location: Bangalore

What is the purpose of robots.txt file?

Postby siva s » Mon Nov 27, 2017 1:27 pm

What is the purpose of robots.txt file?

Posts: 26
Joined: Wed Sep 28, 2016 11:09 am
Location: Dubai

Re: What is the purpose of robots.txt file?

Postby johnwiliam5946 » Mon Nov 27, 2017 1:34 pm

Robots.txt is one way of telling the Search Engine Bots about the web pages on your website which you do not want them to visit.
Robots.txt is useful for preventing the indexation of the parts of any online content that website owners do not want to display. It is important to decide the location of Robots.txt very carefully, or else errors might occur while displaying the website.

Posts: 79
Joined: Mon Mar 27, 2017 4:16 am

Re: What is the purpose of robots.txt file?

Postby sinelogixtech » Tue Nov 28, 2017 5:01 am

Robots.txt is file which is very important for a website, as there are having many important files and web pages which we don t want to crawl in Google and don t want that users can enter in these page.
To exclude this pages from crawling from Google, we use or create the robot.txt file in SEO.

Posts: 114
Joined: Sun Jun 01, 2014 8:24 am
Location: India

Re: What is the purpose of robots.txt file?

Postby seojesica » Tue Nov 28, 2017 6:42 am

It represents to various type of bots to which pages you allow or disallow according to your choices.

Posts: 4
Joined: Tue Nov 28, 2017 12:17 pm

Re: What is the purpose of robots.txt file?

Postby aprilrino » Tue Nov 28, 2017 12:25 pm

Robots.txt is a content record website admins make to teach web robots (normally web index robots) how to slither pages on their site. The robots.txt record is a piece of the robots rejection convention (REP), a gathering of web norms that manage how robots creep the web, access and file substance, and serve that substance up to clients. The REP additionally incorporates orders like meta robots, and page-, subdirectory-, or vast guidelines for how web search tools should treat joins, (for example, "take after" or "nofollow").

By and by, robots.txt documents show whether certain client operators (web-creeping programming) can or can't slither parts of a site. These slither guidelines are indicated by "denying" or "permitting" the conduct of certain (or all) client specialists.

Posts: 44
Joined: Thu Jun 22, 2017 5:31 am
Location: united states

Re: What is the purpose of robots.txt file?

Postby andreausa » Mon Dec 04, 2017 10:41 am

A robots.txt file gives instructions to web robots about the pages the website owner doesn’t wish to be ‘crawled’. For instance, if you didn’t want your images to be listed by Google and other search engines, you’d block them using your robots.txt file.

Posts: 4
Joined: Fri Dec 01, 2017 10:15 am
Location: Mumbai

Re: What is the purpose of robots.txt file?

Postby rachana » Mon Dec 04, 2017 11:17 am

Robots.txt : It is a text file created by webmasters that instruct web robots how to crawl pages in their website.
The main purpose of this file is to indicate whether the pages of their website should be crawled or not. The crawl instructions are “allowing” or “disallowing” which are specified by user agents.

Posts: 8
Joined: Thu Dec 07, 2017 4:12 am

Re: What is the purpose of robots.txt file?

Postby sirishasiri » Thu Dec 07, 2017 4:26 am

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

Posts: 19
Joined: Thu Jul 21, 2016 1:26 pm

Re: What is the purpose of robots.txt file?

Postby jackluter » Thu Dec 07, 2017 11:38 am

Robot.txt file is the important for the websites, so if you have any private are secure pages in your website you are going tell for google through robot.txt file don’t crawl that information it is easy to understand. And you have to hide information from crawlers and robots and bots.

Return to SEO General

Who is online

Users browsing this forum: No registered users and 3 guests