Hacker Newsnew | past | comments | ask | show | jobs | submit | more BiteCode_dev's commentslogin

I worked with a client that implemented their own Python version of this to deduplicate citizen entries in a big french gov database. It worked well.

Of course, nowaday I would probably just tell them to use datasketch (https://pypi.org/project/datasketch/).

With this trip to memory lane, I looked around a little, and noticed people are still creating new stuff on the topic. E.G:

https://pypi.org/project/rensa/

Which is basically a more specialized but faster version of datasketch minhash, written in rust, with a little python on top.


For deduplicating people, the Fellegi Sunter model is also a powerful approach. Splink[0] is a free Python library that implements this for big datasets. Probably you could combine parts of both approaches as well. Full disclosure: I am the lead author.

I've also written up some interactive tutorials on how the method works [1] if anyone's interested

[0]https://github.com/moj-analytical-services/splink [1]https://www.robinlinacre.com/intro_to_probabilistic_linkage/


Wonderful library and a really great set of explainers! Thanks for sharing this. Wish I had this for quite a few past projects…


Interesting.

What would you say are the main differences with other approaches?


The Fellegi Sunter model is able to estimate the importance of different types of information from the data itself (i.e. unsupervised learning).

For instance, a match on a date of birth column lends a greater weight of evidence in favour of a match than a match on first name (since dob has higher cardinality).

The method is also able to estimate weights for fuzzy matches (how much evidence in favour of a match is close match on dob with one character difference), and also how much evidence against a match a mismatch is.

For instance, if you have very high data quality on gender, then a match on gender doesn't tell you much, but a mismatch on gender is quite strong evidence against the idea two records match.

I have a blog post here that delves into this a bit more: https://www.robinlinacre.com/fellegi_sunter_accuracy/


It doesn't sound like the approaches are incompatible. You can use minhash LSH to search a large set and get a top-k list for any individual, then use a weighted average with penalty rules to decide which of those qualifies as a dupe or not. Weighted minhash can also be used to efficiently add repeats to give some additional weighting.


Ok, so you get better accuracy if you datasets have obviously weighted fields. Do you pay that in perfs, and if yes how much?


The performance is pretty good because the prediction is ultimately just adding up match weights. Much of the performance is dictated by

(1) The blocking approach you choose (how wide you cast the net in searching for matches. This is actually somewhere were minhash can be used in conjunction

(2) whether you choose to use complex fuzzy matching functions and how many - this is chosen by the user.

There's some benchmarking results here: https://www.robinlinacre.com/fast_deduplication/

Overall it's an approach which makes a pretty good trade off between speed and accuracy. That's why it's used by many national stats institutes (UK, US, Aus, Germany etc.) - because it's capable of working on country-population sized datasets.


There is also gaoya ( I am the author ), which is also written in Rust and has python bindings. datasketch is great, but it is not performant enough for my use case. gaoya is used in a production system for large scale clustering https://github.com/serega/gaoya


Thanks, 10 years of HN and I was still using "site:news.ycombinator.com"


Try https://hn.algolia.com — it’s great


That's where the footer search field points.


If you go on Tik Tok, sometimes you see fake French news, with the proper setting, format, voices, and sometimes faces.

It's quite believable, especially on a platform where you scroll mindlessly.

But if you look for the bit online, you will not find it on the official TV channel website because it never happened.

I'm used to my gov and American propaganda, but they are rarely faking the whole thing, mostly selectively presenting a certain truth or point of view.

That's new to me: completely fabricated, sophisticated facts and media.


You can be critics of both Russia and America.

We have cancer and you want to ignore it because we also have diabetes, that's a weird stance.

As a member of the EU, it's only logical to be more worried about Russia crossing the Ukraine border than American propaganda.

The risk the US lose their head and decide to attack the EU is nil.

The risk Russia decides that they want to restore the USRR if the Ukraine invasion is a success is plausible.

Is it going to happen? We don't know.

But we don't want to find out.


As another EU member I can say as well that I'd prefer any US hegemony (financial, cultural, etc with all their negative effects) over the physical violence and authoritarian regime coming from the East.


Exported in WASM to play directly in the browser, including Horn of the Abyss.

I would pay for this.

You'd probably have to redo the art though, since 3DO will unlikely do this port and I doubt they would licence the IP.


> I doubt they would licence the IP

But would they go after you for using it.


Underrated book IMO.

I enjoyed it more than the 3 body problem: the characters are better written, their motivations make more sense despite having much less time for actually developing them.

I was really having fun trying to figure out how to make the best of an impossible to solve situation. And the SCP lore is wonderful.


I haven't read _Their Is No [...]_, but I have read _Fine Structure_, another book by QNTM. It's an exhilarating read with some forgivable issues and I'd heartily recommend it to anyone interested in heady sci-fi.

I would never recommend 3 Body Problem. I found it and its sequel to be irredeemably unpleasant.


More than that, it takes a very sophisticated and well-integrated society to no only make it possible, but affordable and profitable.

With the tech and scales of the times, it's mind-blowing and a testament to how badass humanity already was.

What could they have achieved with what we have today?


> With the tech and scales of the times, it's mind-blowing and a testament to how badass humanity already was.

At the scale of the Homo Sapiens Sapiens, Antiquity is yesterday. A good rule of thumb in History is that people were and will be exactly the same as we are.


We feel the same about mushroom spots in France.

There is a saying: during the german occupation, people betrayed friends, and nation, but they shut their mouths about the mushroom spots.


Even if it takes 10 minutes per ToS update per service, that's still a good 100 of hours in your entire life (this they don't provide diffs) just to do that.

It should be illegal.


> Even if it takes 10 minutes per ToS update per service, that's still a good 100 of hours in your entire life

That would mean 600 TOS, which means you’re not following points 1 and 2. Furthermore, as soon as you find anything objectionable, you stop reading. After the first couple of them you don’t need 10 minutes.

> It should be illegal.

There have been instances of unenforceable TOS, and some countries are pushing for contracts to have mandatory summaries of each section. Find out about it in your country and see how you can help.


I remember that someone made a video reading the amazon TOS. It was 9 hours long I believe. That's without stopping, sleeping, eating, and assuming you understand every single sentence.

TOS should be illegal.

Imagine if you had to agree to ToS while entering at Cosco.

Or if you actually had to read 9hours of a legal doc to shop at wallmart.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: