Some web sites see huge crawl spikes from Google

Glenn Gabe noted that he’s been observing some strange Google bot activity recently and said he’s seeing “crawling spikes across a number of websites” beginning on January 12-13. I then checked some of the properties and found that I see the same thing on some, not most.

To be clear this doesn’t apply to all websites, in fact it seems to apply to less than 10% of the properties that I personally have access to. But it’s unusual behavior on the part of Google. You can observe this here and there on a website, especially when there are major technical changes to the website. But you wouldn’t normally see it unless there are big changes like this.

Here’s one of the screenshots Glenn shared Twitter:

Glenn Gabe added, “For most spikes, they fall under ‘Update’ vs. ‘Discover’, which makes sense… but is worth mentioning nonetheless. Both smartphone and desktop tops for some.”

Others see it too:

And just to be clear, I’m not saying that’s a *bad* thing. It’s just interesting to see massive crawl spikes across a number of sites all starting on the same day. 🙂 Would be interesting to hear if Google made any changes there from a crawling point of view.

— Glenn Gabe (@glenngabe) January 18, 2022

I’ve seen this for a minority of properties I have access to, but yeah, super weird pic.twitter.com/9SQqGtWTJ6

— Barry Schwartz (@rustybrick) January 18, 2022

I noticed that as well. Interestingly (and somewhat underestimated) I noticed for one website that 25% of the Googlebot type was “page resource load”. I’ve never seen it higher than 8%, usually around 3%-4%. Any idea why that might be so high Glenn? pic.twitter.com/BBOKeb8Odq

— Chris Ainsworth (@chrisains) January 18, 2022

You can check yours Crawl stats report over here and let us know if you’ve seen this tip too?

Just a warning, Google said so Large crawl rate increases are not related to algorithm updates, although, sometimes it may be – could be.

forum discussion below Twitter.