Register

What is the limit of a robot.txt file?

Basic discussions about SEO
Posts: 579
Joined: Sat Nov 09, 2013 2:26 pm
Location: Bangalore

What is the limit of a robot.txt file?

Postby siva s » Thu Oct 04, 2018 4:52 am

What is the limit of a robot.txt file?

Posts: 172
Joined: Mon Mar 27, 2017 4:16 am

Re: What is the limit of a robot.txt file?

Postby sinelogixtech » Thu Oct 04, 2018 6:58 am

Yes, googlebot reads the first 500 KB present in the robots.txt file.

User avatar
Posts: 754
Joined: Mon Jan 18, 2016 5:57 pm
Location: Forum

Re: What is the limit of a robot.txt file?

Postby RH-Calvin » Mon Oct 08, 2018 6:52 am

There is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out. A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb.
Cheapest VPS | $1 VPS Hosting
Cheap Dedicated Servers | Free Setup and IPMI | Unmetered Bandwidth

Posts: 21
Joined: Thu Sep 20, 2018 10:43 am

Re: What is the limit of a robot.txt file?

Postby Ethon Johnson » Tue Oct 09, 2018 7:15 am

A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500 kilobytes (KB)

Posts: 115
Joined: Thu Sep 08, 2016 10:25 am
Location: Derby

Re: What is the limit of a robot.txt file?

Postby Mangat Singh Dhiman » Mon Oct 29, 2018 12:16 pm

The limit of robots.txt file in terms of filesize 500 KB .

Posts: 34
Joined: Thu Jul 21, 2016 1:26 pm

Re: What is the limit of a robot.txt file?

Postby jackluter » Mon Nov 12, 2018 12:22 pm

there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out.

Posts: 20
Joined: Fri Nov 23, 2018 1:06 pm

Re: What is the limit of a robot.txt file?

Postby yuva12 » Mon Dec 03, 2018 8:07 am

Your robots.txt file must be smaller than 500KB.

Posts: 24
Joined: Fri Nov 16, 2018 9:23 am

Re: What is the limit of a robot.txt file?

Postby jonathan brown » Fri Dec 14, 2018 7:13 am

Hi,

Yes, the limitations are:

1. The google bot can only read-only first 500KB robot.txt file data.
2. The robot.txt file needs to be added on the top of the domain of your website so that its command gets easily sharable with required details.
3.No assurance that a crawler or robot will visit your site and use your robots.txt file. Actually, search engines use and obey robots.txt file, but malware robots will not.
4. Sometimes the robot.txt file elements get ignored by some search engines

Return to SEO General

Who is online

Users browsing this forum: No registered users and 2 guests