Google deduplication of prime tales from internet search outcomes

We know Google has deduplication Efforts to determine when a featured snippet is displayed so that it may be Remove the snippet from the main web results. Google can do the same in some cases when a URL is listed in the Top Stories section and then doesn’t appear in the main web results.

In some cases, Google may remove the web search result snippet when showing the same result in the top story section of Google search results.

Danny Sullivan of Google further said so Twitter following a complaint from The Verge. Danny said, “It shows up in the top stories, it’s deduplicated from the rest of the site. Deduplication can often be useful. Do this search the way the user might do it, using solution-seeking terms instead of uncommon terms in the headline You’re at the top of Top Stories, and deduplication means there’s more variety from other publications. During such searches, our systems also generally attempt to display the most helpful and reliable information they can find. That’s why you don’t see a lot of duplicates of your item appearing. Duplicates certainly exist, but showing them isn’t that helpful. This leads to headline-oriented searches. As I said before, this is very common among writers. I used to always do this, myself. But headline searches usually contain a lot of terms, so our systems switch to pages that contain those terms. This means that authors are more likely to find duplicates, although these are different from how they appear in typical searches that readers would perform. But our deduplication feature can work for those as well, as it did in this case. Again, deduplication can help. But we also understand the concern this could raise. We’ve been doing this with Top Stories since last May, but we’ll review this again to see if we should proceed or maybe make other changes. Also, I’m still checking it out, but I think this deduplication is particularly unique in that on Top Stories it only occurs when a single story is shown, or maybe just for the very first story shown.”

So you can see Danny citing this as an excuse for why other publications are showing up in the web results and not The Verge for this query. But as you can also see, he seems to explain that sometimes it doesn’t work that way.

Deduplication can often be useful. If you do this search the way the user would, using solution-seeking terms instead of uncommon terms in the headline, you’ll be up there in the Top Stories, and deduplication means there’s more variety from other releases …. pic.twitter.com/638IAZLWIV

— Danny Sullivan (@dannysullivan) January 18, 2022

The interesting part is that Google actually deduplicates here when it comes to top stories.

The funny thing is that hours ago, News SEO Barry Adams created an entire Twitter thread about this:

That’s not the case. It can easily be disproved; Look for recent articles published in a major news outlet (BBC, The Guardian, NYT, etc.). Search for relevant keywords about this article.

Chances are you will see the article in both the message box and the regular SERP. pic.twitter.com/3u7HH7jppq

— Barry Adams 📰 (@badams) January 18, 2022

So a newly published article had very little opportunity to rank in regular SERPs. It can show up in Top Stories boxes, but it can take days, weeks, or even months to rank in regular SERPs.

Also, authoritative publishers have an advantage here because they are authoritative.

— Barry Adams 📰 (@badams) January 18, 2022

In total; There is no “News” filter in Google SERPs. Articles can and do appear in both Top Stories and regular SERPs.

But often this doesn’t happen due to issues of speed, intent and competition.

/End

— Barry Adams 📰 (@badams) January 18, 2022

So I pointed out Danny’s tweet to Barry Adams and his response:

Color me shocked Danny was wrong about something. 🤪

— Barry Adams 📰 (@badams) January 18, 2022

So I did some sample searches and Google seems to be removing a recent story from this site from the top stories and showing it in the web results while also removing those stories from the web results in the top stories. Note that here SER is not in the top stories but in the web results and at the same time SEL and SEJ are in the top stories but not in the web results (click to enlarge):

But yes, the story is displayed in the News tab:

But sometimes Google doesn’t deduplicate these top stories, a search query for [the hidden resignation] shows the Business Insider story in Top Stories twice in some browsers and once in some (thanks Glenn Gabe for asking and discussing with me):

Deduplicated:

Not Deduplicated:

Here’s another example from Glenn:

Another good example of a URL showing up in Top Stories and also ranking in the 10 blue links. But from what Danny explained yesterday, Google’s deduplication system for Top Stories is nuanced. So it won’t happen all the time (which makes sense based on other queries). pic.twitter.com/ywiwiQnEfw

— Glenn Gabe (@glenngabe) January 19, 2022

So it’s not clear when and why Google might remove a URL from displaying in web search if it also appears in top stories. It could be a timing issue or something else.

Danny Sullivan said, “Also, I’m still checking, but I think this deduplication is particularly unique in that on Top Stories it only occurs when a single story is shown, or maybe just for the very first story shown.”

Also, I’m still checking, but I think this deduplication is particularly unique in that on Top Stories it only occurs when a single story is shown, or maybe just for the very first story shown.

— Danny Sullivan (@dannysullivan) January 18, 2022

We hope for clarity soon.

forum discussion below Twitter.