Does it work? Maybe, maybe not. Perhaps you get some fantastic candidates (true positives), and perhaps you get people that are very god at gaming the system (false positives).
The process, as it is, is kinda like trying to select potential mathematicians on the basis of how well they solve HS AP Math problems. It is fully possible to rote learn every kind of integral and derivative under the sun, if you just solve enough - without actually understanding the underlying principles. It becomes a pattern recognition problem.
There are tons of anecdotes from seemingly false positives, when it comes to tech hiring. The web is filled with "I was very lucky, because they re-used problems I had just solved".
BTW, when I say false positives, I don't mean incompetent programmers / engineers - I just refer to those that do not master the subjects they're being tested on, but average candidates (on the subject) that luck out on getting asked the right question.
I still think Goodhart's law stands true for this trend.
People game the system, because they want to earn more money. Companies make the system more rigorous and robust against gaming. People still find ways to game the system, and it essentially becomes a race to the bottom. Along the way you start losing out on terrific candidates, because they refuse to partake in the increasing demands. So you end up with a mixture of very talented engineers, and very able test-takers.
Does it work? Maybe, maybe not. Perhaps you get some fantastic candidates (true positives), and perhaps you get people that are very god at gaming the system (false positives).
The process, as it is, is kinda like trying to select potential mathematicians on the basis of how well they solve HS AP Math problems. It is fully possible to rote learn every kind of integral and derivative under the sun, if you just solve enough - without actually understanding the underlying principles. It becomes a pattern recognition problem.
There are tons of anecdotes from seemingly false positives, when it comes to tech hiring. The web is filled with "I was very lucky, because they re-used problems I had just solved".
BTW, when I say false positives, I don't mean incompetent programmers / engineers - I just refer to those that do not master the subjects they're being tested on, but average candidates (on the subject) that luck out on getting asked the right question.
I still think Goodhart's law stands true for this trend.
People game the system, because they want to earn more money. Companies make the system more rigorous and robust against gaming. People still find ways to game the system, and it essentially becomes a race to the bottom. Along the way you start losing out on terrific candidates, because they refuse to partake in the increasing demands. So you end up with a mixture of very talented engineers, and very able test-takers.