Register

What is the limit of a robot.txt file?

Basic discussions about SEO
Posts: 570
Joined: Sat Nov 09, 2013 2:26 pm
Location: Bangalore

What is the limit of a robot.txt file?

Postby siva s » Thu Oct 04, 2018 4:52 am

What is the limit of a robot.txt file?

Posts: 172
Joined: Mon Mar 27, 2017 4:16 am

Re: What is the limit of a robot.txt file?

Postby sinelogixtech » Thu Oct 04, 2018 6:58 am

Yes, googlebot reads the first 500 KB present in the robots.txt file.

User avatar
Posts: 732
Joined: Mon Jan 18, 2016 5:57 pm
Location: Forum

Re: What is the limit of a robot.txt file?

Postby RH-Calvin » Mon Oct 08, 2018 6:52 am

There is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out. A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb.
Cheapest VPS | $1 VPS Hosting
Cheap Dedicated Servers | Free Setup and IPMI | Unmetered Bandwidth

Posts: 21
Joined: Thu Sep 20, 2018 10:43 am

Re: What is the limit of a robot.txt file?

Postby Ethon Johnson » Tue Oct 09, 2018 7:15 am

A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500 kilobytes (KB)

Posts: 111
Joined: Thu Sep 08, 2016 10:25 am
Location: Derby

Re: What is the limit of a robot.txt file?

Postby Mangat Singh Dhiman » Mon Oct 29, 2018 12:16 pm

The limit of robots.txt file in terms of filesize 500 KB .

Posts: 34
Joined: Thu Jul 21, 2016 1:26 pm

Re: What is the limit of a robot.txt file?

Postby jackluter » Mon Nov 12, 2018 12:22 pm

there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out.

Return to SEO General

Who is online

Users browsing this forum: Google Adsense [Bot], jonathan brown and 1 guest