For deduplicating people, the Fellegi Sunter model is also a powerful approach. Splink[0] is a free Python library that implements this for big datasets. Probably you could combine parts of both approaches as well. Full disclosure: I am the lead author.
I've also written up some interactive tutorials on how the method works [1] if anyone's interested
The Fellegi Sunter model is able to estimate the importance of different types of information from the data itself (i.e. unsupervised learning).
For instance, a match on a date of birth column lends a greater weight of evidence in favour of a match than a match on first name (since dob has higher cardinality).
The method is also able to estimate weights for fuzzy matches (how much evidence in favour of a match is close match on dob with one character difference), and also how much evidence against a match a mismatch is.
For instance, if you have very high data quality on gender, then a match on gender doesn't tell you much, but a mismatch on gender is quite strong evidence against the idea two records match.
It doesn't sound like the approaches are incompatible. You can use minhash LSH to search a large set and get a top-k list for any individual, then use a weighted average with penalty rules to decide which of those qualifies as a dupe or not. Weighted minhash can also be used to efficiently add repeats to give some additional weighting.
The performance is pretty good because the prediction is ultimately just adding up match weights. Much of the performance is dictated by
(1) The blocking approach you choose (how wide you cast the net in searching for matches. This is actually somewhere were minhash can be used in conjunction
(2) whether you choose to use complex fuzzy matching functions and how many - this is chosen by the user.
Overall it's an approach which makes a pretty good trade off between speed and accuracy. That's why it's used by many national stats institutes (UK, US, Aus, Germany etc.) - because it's capable of working on country-population sized datasets.
There is also gaoya ( I am the author ), which is also written in Rust and has python bindings. datasketch is great, but it is not performant enough for my use case. gaoya is used in a production system for large scale clustering
https://github.com/serega/gaoya
If you go on Tik Tok, sometimes you see fake French news, with the proper setting, format, voices, and sometimes faces.
It's quite believable, especially on a platform where you scroll mindlessly.
But if you look for the bit online, you will not find it on the official TV channel website because it never happened.
I'm used to my gov and American propaganda, but they are rarely faking the whole thing, mostly selectively presenting a certain truth or point of view.
That's new to me: completely fabricated, sophisticated facts and media.
As another EU member I can say as well that I'd prefer any US hegemony (financial, cultural, etc with all their negative effects) over the physical violence and authoritarian regime coming from the East.
I enjoyed it more than the 3 body problem: the characters are better written, their motivations make more sense despite having much less time for actually developing them.
I was really having fun trying to figure out how to make the best of an impossible to solve situation. And the SCP lore is wonderful.
I haven't read _Their Is No [...]_, but I have read _Fine Structure_, another book by QNTM. It's an exhilarating read with some forgivable issues and I'd heartily recommend it to anyone interested in heady sci-fi.
I would never recommend 3 Body Problem. I found it and its sequel to be irredeemably unpleasant.
> With the tech and scales of the times, it's mind-blowing and a testament to how badass humanity already was.
At the scale of the Homo Sapiens Sapiens, Antiquity is yesterday. A good rule of thumb in History is that people were and will be exactly the same as we are.
Even if it takes 10 minutes per ToS update per service, that's still a good 100 of hours in your entire life (this they don't provide diffs) just to do that.
> Even if it takes 10 minutes per ToS update per service, that's still a good 100 of hours in your entire life
That would mean 600 TOS, which means you’re not following points 1 and 2. Furthermore, as soon as you find anything objectionable, you stop reading. After the first couple of them you don’t need 10 minutes.
> It should be illegal.
There have been instances of unenforceable TOS, and some countries are pushing for contracts to have mandatory summaries of each section. Find out about it in your country and see how you can help.
I remember that someone made a video reading the amazon TOS. It was 9 hours long I believe. That's without stopping, sleeping, eating, and assuming you understand every single sentence.
TOS should be illegal.
Imagine if you had to agree to ToS while entering at Cosco.
Or if you actually had to read 9hours of a legal doc to shop at wallmart.
Of course, nowaday I would probably just tell them to use datasketch (https://pypi.org/project/datasketch/).
With this trip to memory lane, I looked around a little, and noticed people are still creating new stuff on the topic. E.G:
https://pypi.org/project/rensa/
Which is basically a more specialized but faster version of datasketch minhash, written in rust, with a little python on top.