What bothers me is what seems to be some sort of psychological distinction between "algorithm" and "process".
The cops* have had processes for years, some of which are sensible and useful and some of which are seemingly sensible but terrible, and some of which are nonsense. Over time, one hopes, the worst of the processes (or the worst steps in some processes) get winnowed out and/or replaced.
Also machinery is integrated into processes and over time its efficacy is evaluated ("shot spotters": worse than useless; police in patrol cars: better in some ways, much worse in others).
But somehow when a mechanically computed algorithm is used as a step in a process, all of this sort of reasoning goes out the window and now it's scary.
> But somehow when a mechanically computed algorithm is used as a step in a process, all of this sort of reasoning goes out the window and now it's scary
Humans following procedure can be analyzed in meaningful ways by non-technical third parties to figure out where a failure point is
Black box program algorithms cannot. They require experts to analyze, who are also often the people who own the algorithm and wrote it in the first place
These algos are part of a process, so subject to the same kind of treatment. The car was a "black box" as far as its social impact when introduced, yet we managed to figure it out.
And a lot of forensic techniques, like fingerprints, flame spreading, graphology and others are epistemologically less sound than the kind of algo discussed in this article yet people seem comfortable with them. At least phrenology has been abandoned (AFAIK...eek)
The cops* have had processes for years, some of which are sensible and useful and some of which are seemingly sensible but terrible, and some of which are nonsense. Over time, one hopes, the worst of the processes (or the worst steps in some processes) get winnowed out and/or replaced.
Also machinery is integrated into processes and over time its efficacy is evaluated ("shot spotters": worse than useless; police in patrol cars: better in some ways, much worse in others).
But somehow when a mechanically computed algorithm is used as a step in a process, all of this sort of reasoning goes out the window and now it's scary.
* and everybody else