Google Can Crawl Sections Of Your Website Extra Steadily & Decide High quality Otherwise Additionally

Google’s Gary Illyes said on the last Search Off the Record podcast that Google can crawl certain sections of your site more frequently and also infer the quality of certain sections of your site differently.

This came up at the 9:09 minute mark into the podcast but Glenn Gabe summarized it super well on Twitter. Glenn said “Google can infer from a site overall which areas they might need to crawl more frequently. Eg if there’s a blog subdirectory & there are signals that it’s popular/important, then Google might want to crawl there more.” “And it’s not just update frequency, it’s also about quality. Eg if G sees a certain pattern is popular (folder), & people are talking about it & linking to it, then it’s a signal that ppl like that directory,” he added .

Here is the video embed:

Here is the transcript of this section on crawl frequency by section of the site:

yes Because like we said, we don’t have infinite space, so we want to index stuff that we think– well, not we– but our algorithms determine that it might be searched for at some point, and if we don’t have signals, for example, yet, about a certain site or a certain URL or whatever, then how would we know that we need to crawl that for indexing?

And some things you can infer from– for example, if you launch a new blog on your main site, for example, and you have a new /blog subdirectory, for example, then we can sort of infer, based on the whole site , whether we want to crawl a lot from that /blog or not.

Then here is the section on quality:

But it’s not just update frequency. It’s also the quality signals that the main site has.

So, for example, if we see that a certain pattern is very popular on the internet, like a slash product is very popular on the internet, and people on Reddit are talking about it, other sites are linking to URLs in that pattern, then it’s a signal for us that people like the site in general.

While if you have something that people are not linking to, and then you are trying to launch a new directory, it’s like, well, people don’t like the site, then why would we crawl this new directory that you just launched?

Forum discussion at Twitter.