Register

What is the limit of a robot.txt file?

Basic discussions about SEO
Posts: 562
Joined: Sat Nov 09, 2013 2:26 pm
Location: Bangalore

What is the limit of a robot.txt file?

Postby siva s » Thu Oct 04, 2018 4:52 am

What is the limit of a robot.txt file?

Posts: 172
Joined: Mon Mar 27, 2017 4:16 am

Re: What is the limit of a robot.txt file?

Postby sinelogixtech » Thu Oct 04, 2018 6:58 am

Yes, googlebot reads the first 500 KB present in the robots.txt file.

User avatar
Posts: 720
Joined: Mon Jan 18, 2016 5:57 pm
Location: Forum

Re: What is the limit of a robot.txt file?

Postby RH-Calvin » Mon Oct 08, 2018 6:52 am

There is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out. A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb.
Cheapest VPS | $1 VPS Hosting
Cheap Dedicated Servers | Free Setup and IPMI | Unmetered Bandwidth

Posts: 18
Joined: Thu Sep 20, 2018 10:43 am

Re: What is the limit of a robot.txt file?

Postby Ethon Johnson » Tue Oct 09, 2018 7:15 am

A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500 kilobytes (KB)

Return to SEO General

Who is online

Users browsing this forum: No registered users and 1 guest