Gary Illyes posted a new blog post on the Google Search Central site asking all of you to stop using 403 and 404 server status codes to reduce the crawl rate of Googlebot. He said they have seen an uptick in the number of sites and CDNs doing this and they need to cut it […]
As you know, Google can now crawl from locations outside of the United States. We know this is rare, it is used in specific situations. But if you want to know if Google is crawling your site from outside of the US, then you need to check your log files. You cannot use the crawl […]
Google has added a new section to the crawl budget management help document. The new section is for the If-Modified-Since request header. The section is titled “Specify content changes with HTTP status codes.” Google wrote“Google generally supports the If-Modified-Since and If-None-Match HTTP request headers for crawling. Google’s crawlers don’t send the headers with all crawl […]
Gary Illyes from Google announced at the Google Search Central Live in Singapore yesterday that Googlebot, Google, can now crawl from outside of the US. He said, according to reports, that the system to handle this is currently live but may not be used in all cases or it may not be used at all […]
There is a fun Twitter thread where Google’s John Mueller and SEO Elmer Boutin talk about expensive pages for Google to crawl. John starts off by explaining that Google doesn’t think about it from an expense or not, it is more about if the page is something that is relevant and useful – that is […]
Google’s Gary Illyes said on the last Search Off the Record podcast that Google can crawl certain sections of your site more frequently and also infer the quality of certain sections of your site differently. This came up at the 9:09 minute mark into the podcast but Glenn Gabe summarized it super well on Twitter. […]
Google also updated the crawl stats report help documentation last week. Google made a lot of support documentation changes in the past week including product rich results, Googlebot doc, job posting help doc, moving the HTTP status codes doc and also this crawl stats report help document. You can see this crawl stats help document […]
Google Search Console’s crawl stats report has a data gap, a missing day, of metrics again. This has happened numerous times, often this is fixed by itself after a few days. So I would check back in a few days – I doubt Google is missing data, it just has a processing issue and hopefully […]
As you know, last week, on April 26th, Google disables the URL parameter tool within Google Search Console and stopped respecting any of the rules added to that tool. Some are asking if the recent crawl spikes they saw with Googlebot was related to this change. Dave Smart posted on Twitter charts showing his crawl […]
The other day I reported again how blocking US based users would require you to also block Googlebot, Google’s crawler. And that would be devastating to a site’s search rankings in Google. But John Mueller of Google said it is not out of the question that Google one day may decide to crawl from outside […]