Google says url case additionally issues for Robots.txt

We have known for likes Google does it and can treat the same url but differently with different cases. Therefore, domain.com/Apple vs. domain.com/apple can be viewed as different URLs by Google. But Google seems to be stricter on this rule when it comes to the robots.txt file.

Using domain.com/Apple vs. domain.com/apple, Google can tell if the pages are the same and canonicalize the URL to one of them so that neither will show up in search and Google may consolidate the signals as well.

But if you have a robots.txt directive for domain.com/apple but not domain.com/Apple, maybe Google is not using a robots directive?

Google’s John Mueller posted a URL case-sensitive video and at the 1:08 mark said, “Another place that the exact URL matters is in the robots.txt file. In the robots .txt file lets you specify which parts of a website shouldn’t be crawled. The robots.txt file also uses exact URLs, so if you have entries there that refer to a version of a URL, they don’t apply to other versions of this url. ” John added that “it is rare for this to cause problems”.

So keep this in mind not only for the general way that Google can canonicalize your URLs, but also for how it handles robots.txt.

Forum discussion at WebmasterWorld and Twitter.