I was once asked to use machine learning to make a record linkage system for some crappy dataset. I got no requirements, of course, so I set it up to have a reasonable balance of precision and recall. After all, the point of asking for an ML system must be to allow fuzzy matches that a simple exact matching system would miss, right?
But my boss apparently got complaints about bad matches, so he changed it to allow exact matches only.
The machine learning system ended up being a Rube Goldberg machine for linking people based on exact name match.
Anecdotally this is pretty typical of the evolution of mo systems.
Some heuristic/standard algorithm works pretty well, but people see cases it didn't catch and think it could be better. An engineer/scientist takes a look and realizes the extra features required need a more complex/ml algorithm to add support for. Years later a model has made it to production, but there are now complaints about too lenient matching.
In my experience, ML works best in scenarios where there is either such immense data volume that .2% improvement is a real benefit (these are rare) or the very notion of a heuristic method simply wouldn’t work.
But my boss apparently got complaints about bad matches, so he changed it to allow exact matches only.
The machine learning system ended up being a Rube Goldberg machine for linking people based on exact name match.