Google On The Most Costly Pages To Crawl

There is a fun Twitter thread where Google’s John Mueller and SEO Elmer Boutin talk about expensive pages for Google to crawl. John starts off by explaining that Google doesn’t think about it from an expense or not, it is more about if the page is something that is relevant and useful – that is what Google cares about.

John goes on to say that the “most “expensive” pages are those we’ve been crawling & indexing for years, and nobody has looked for them ever.” But Google crawls and indexes those pages because “what if tomorrow is different? It’s good to be prepared,” he added. He then threw out the line “15% of all searches are new every day, you never know.” Indeed.

Here are those tweets:

All of web search costs time & resources. I don’t think it’s useful to compare the processing effort of specific media types, the important part is that relevant & useful content can be linked to for users who are looking for it.

— John Mueller is mostly not here 🐀 (@JohnMu) November 15, 2022

Thank you for the response. You bring up a very good point that I believe we SEOs talk about often, but our advice is not heeded as much as it should be.

— Elmer Boutin (@rehor) November 15, 2022

Sure, Google tries to be as efficient as possible when it comes to everything it does. That means discovering, crawling, indexing, ranking and serving are all done with efficiency in mind. But not at the expense of not providing the most useful and relevant search result to the searcher.

Forum discussion at Twitter.