I agree, the statistical analysis in the original post makes me very uneasy. I think it could be a case where the conclusion is correct, even though argument isn't necessarily.
And yes, the fact that the slope is less than one is fairly uninteresting.
The real problem here is that the Dunning-Kruger effect, as it's classically stated, claims that if you asked four people to rank themselves in terms of ability, the result would be 1-3-2-4, ie the people who know a little would put themselves above the people who know a lot but aren't quite experts. The problem is the data shows that they'd actually rank themselves correctly 1-2-3-4. But such a boring finding probably wouldn't have made the authors quite as famous, which might be why they tried bit of data mangling, and they found this really cool story that everyone would secretly love to be true.
Which is a shame, because I think the fact that the mean of perceived ability is too high (and the variance too low) is really interesting too, and perfectly supported by the raw data.
Yes. The methodology in the original D&K is quite shoddy, and vulnerable to e.g. good old regression to the mean, and the interpretations are too strong. This is sadly very common in psychology (and many other fields I'd guess) and even researchers don't care so much if the story is juicy enough.
The pop version of the DK effect seems to be something like a 4-3-2-1 ranking, which is obviously not supported by the data.
But they wouldn't. They'd rank themselves something like 1,2,2,3. We're not dealing with a population collaborating to all rank themselves in order, but rather each person individually estimating where their abilities lie in the population.
The point is that if you ask someone in the, say, 5th percentile of ability what their ability is compared to the population, they might say 25th percentile. Ask someone at the 25th,and they might say 40th. At the 40th they could say 55th. And at the 90th, maybe they'll say 80th. So yes, if you order their guesses, they will be in roughly the correct order. But, crucially, that doesn't mean that they are ranking themselves correctly!
And yes, the fact that the slope is less than one is fairly uninteresting.
The real problem here is that the Dunning-Kruger effect, as it's classically stated, claims that if you asked four people to rank themselves in terms of ability, the result would be 1-3-2-4, ie the people who know a little would put themselves above the people who know a lot but aren't quite experts. The problem is the data shows that they'd actually rank themselves correctly 1-2-3-4. But such a boring finding probably wouldn't have made the authors quite as famous, which might be why they tried bit of data mangling, and they found this really cool story that everyone would secretly love to be true.
Which is a shame, because I think the fact that the mean of perceived ability is too high (and the variance too low) is really interesting too, and perfectly supported by the raw data.