Trafilatura's Deduplication SimHash could be useful for crawler

[YeonwooSung]

Trafilatura's Deduplication Official Doc

The presence of duplicate content on the web can have detrimental effects on data quality and computational efficiency.

Trafilatura offers two types of duplicate detection: identical segment detection, which works on exact same text parts, and near-duplicate removal, which uses locality-sensitive hashing to identify similar texts.

[YeonwooSung]

Neosearch crawler uses 'deduplicate' option of the trafilatura, which internally uses dedup SimHash stuff.
But, it would be nice to have the utility function for SimHash calculation, so that we could reuse it in other pjts.