Tag Archives: 500KB

Gary Illyes shared a nice little tidbit on LinkedIn about robots.txt files. He said that only a tiny number of robots.txt files are over 500 kilobytes. I mean, most robots.txt files have a few lines of text, so this makes sense but still, it is a nice tidbit of knowledge. Gary looked at over a […]