Tag Archives: crawling

Google Says Disallowing UTM Parameters In URLs Will not Help With Crawling Or Rating

Google’s John Mueller said on Reddit that disallowing URLs with UTM parameters in them won’t help you to improve crawling or rating with Google Search. He added that a site should try to keep its internal URLs clean and consistent, but over time, the canonical tags should help with external links that carry UTM parameters […]

Google No Longer Has A Favicon User Agent & Required Googlebot-Image and Googlebot Crawling

Google has updated its favicon search developer documentation to remove the section for the Google Favicon user agent and to clarify that if you want Google to show your favicon you must allow both Googlebot-Image and Googlebot to crawl your pages. Here is what changed in the favicon documentation – you can also compare yourself […]

Google High quality Updates Can Affect Crawling & Indexing

Google’s John Mueller said again that its quality updates, like core updates and the others in the family, can not just impact the ranking of a page in Google Search but also how Google crawls and if Google decides to index that page. John said on mastodon“a lot of the quality updates that we make […]

Google Once more Says Cache-Control Headers Aren’t Used For Crawling & Indexing Exterior Of Embedded Content material

In 2018, John Mueller of Google said Google does not use cache-control headers when crawling. He said then that the has no impact on GoogleBot and how it crawls your web pages. He said that again today, at the end of 2022, but added, “at most, they might be used in rendering for embedded content.” […]

Google Once more Says Spikes In Crawling Exercise Not A Signal Of The Useful Content material Replace Rollout

Google’s John Mueller was asked if a spike in crawler activity would be a sign that the helpful content update is rolling out. John said no, as expected because he said no to this question numerous times on previous updates. Here are the relevant set of tweets I am referencing here: No — 🖖 johnmu […]

Bizarre Google Algorithm Patterns, New Google Advert Label, Uncommon Googlebot Crawling Situation, Search Console Loss & Extra Messages

The Google algorithm ranking tracking tools seems to be having fun, or maybe they are off, I am seeing mixed signals of Google ranking updates. Google is testing a way more visible Google ad label with a green curved border, shocking. Google had some weird crawling issue last week, they called it a temporary drop […]

Google just isn’t indexing or crawling any new content material this morning

It seems like Google has a major indexing or crawling problem this morning – again. Google doesn’t seem to have added any new content in the last hour or two. You can see that sites like the Wall Street Journal, the New York Times, and niche publishers like TechCrunch and The Verge don’t see their […]

Google is not crawling over HTTP / 3 but

Google’s John Mueller confirmed that the Googlebot is not yet crawling over HTTP / 3. He said that if you implement HTTP / 3 for your site, it doesn’t mean it will not benefit your users and, by extension, possibly your most important web vitals metrics. But the Googlebot isn’t crawling over it yet. That […]

Google Core Replace November Continuation, Search Console Redesign, Crawling Bugs & Google Ads Credit

Happy Thanksgiving – we had another surprise from Google, where the second wave of Google’s core November 2021 update took place on Wednesday and Thursday – yes, the busiest time for online shopping. Previously there was a survey that said 50% of SEOs hadn’t noticed any changes from the core update. Google has introduced a […]

Google typically has completely different crawling budgets / capacities to your photographs / information on a CDN

Google’s John Mueller has confirmed that for websites that host their images (or probably other files) on an external CDN, as we do here, Google will have separate server capacity (perhaps crawl budget) for your domain and the domain of the CDN . Like the image above, it’s hosted on AWS and isn’t on that […]