One of the problems with things like this is how do you know which site has copied from another, especially if you don't want a list of hard-coded exceptions.
Related is if you have a lot of content from git repositories that are mirrored from different locations (GitHub, GitLab, etc.), all of which are showing the same content. Or if different sites are hosting versions of public domain texts. You don't want to derank those results, even if they are similar to the "copy a popular site" websites.
> One of the problems with things like this is how do you know which site has copied from another, especially if you don't want a list of hard-coded exceptions.
The core function of that is actually pretty simple:
1. Strip all X/HTML tags
2. Run `diff`
Sure, it's not perfect, but an organization that pursues academic quantum computing research can sure as hell afford to run the results of the above against an AI to check for similarities.
The issue is not figuring out if site X has a copy of a page or text from site Y. The issue is how do you know which site is the source of the information. In other words, the technical bit is easy but determining what to do with that is hard. Especially if you don't want a lot of hard coded exceptions.
In the case of things like other sites copying from stack overflow, github, etc. you can figure out which is the source of the information.
Lets day you have decided to make github the source of that information, and derank any other sites that also have that information. As a result, you will derank gitlab for having git mirrors of projects, as the source will match that on github. You will also derank sites like lkml as that contains the commit message descriptions, and the patches will partially match content from the kernel source hosted on github.
Lets also say you've decided to make wikipedia and other wikimedia sites the source of information. Congratulations, you've now deranked project gutenberg for hosting the same public domain texts as wikimedia, along with other sites like the official sites for authors like Jack London. Plus any blog that includes portions of these to discuss or analyze them.
> he issue is how do you know which site is the source of the information. In other words, the technical bit is easy but determining what to do with that is hard.
I interviewed with a 3 or so person company that did just that I think.
Google knows instantly how much time you spend on a page and how quickly you return to the search results because it is a useless copy/paste page from SO.
And probably they track content age too? Since they somehow can lookup by content (they can detect duplicated content), there's probably a date remembered too
So they could know if a code snippet was already X years old? And from where, originally. (Unless it got edited a lot but that's not the case here?)
Related is if you have a lot of content from git repositories that are mirrored from different locations (GitHub, GitLab, etc.), all of which are showing the same content. Or if different sites are hosting versions of public domain texts. You don't want to derank those results, even if they are similar to the "copy a popular site" websites.