Google Says Very Few Robots.txt Recordsdata Are Over 500KB

Gary Illyes shared a nice little tidbit on LinkedIn about robots.txt files. He said that only a tiny number of robots.txt files are over 500 kilobytes. I mean, most robots.txt files have a few lines of text, so this makes sense but still, it is a nice tidbit of knowledge.

Gary looked at over a billion robots.txt files that Google Search knows about and said only 7,188 of them were over 500 KiB. That is less than 0.000719%.

He wrote, “One would think that out of the billions (yes, with a 🐝) of robots.txt files Google knows of more than 7188 would be larger in byte size than the 500kiB processing limit. Alas. No.”

Yea, the SEO point here is that Google can process up to 500KB of your robots.txt file but most of those files don’t even come close to that file size.

Forum discussion at LinkedIn.