Tag Archives: Crawlers

SEM & SEO

Google Says Double Slashes In URLs Can Confuse Some Search Crawlers

Posted on by admin
24
Dec

Gary Illyes said some search engines can be confused by seeing double slashes in a URL. Previously, Google suggested not to use double slashes in your URLs but here Gary is saying that technically pages load with double slashes but with some search engines may have issues crawling URLs with double slashes. Gary explained this […]

Continue reading →
SEM & SEO

Google Site Verifier User Agent Added To Google Crawlers

Posted on by admin
21
Sep

Google has added a new Google crawler, a new user agent, to the Google spider list. Its name is Google Site Verifier User Agent. This user agent ignores robots.txt rules, by the way, and I assume it is used for verifying sites with Search Console and other Google tools and services. Google posted the new […]

Continue reading →