I wish something similar existed for databases. I think exact figures would be hard to get, but I believe there are many 2x 10x differences that we should be aware of.
> Benchmarking is hard and a lot of reports are bogus. However they are still very useful for a lot of developers.
In this case we were presented with a benchmark setup that failed to perform the task it supposedly benchmarked. That's not hard to avoid, and it makes the benchmark completely useless and misleading.
I don't think that this problem can be generalized in the sense you do here, since the problem with benchmarks usually isn't a complete failure to perform the task to be benchmarked, but things like finding a set of tests that give a fair representation of what you'd typically use the subjects for, or performing the tasks in idiomatic and optimal ways.
Don't a lot of those benchmarks end up only measuring how fast your language can call out to GMP to do the real work? And regex-dna ends up measuring your regex implementation which for a lot of them is again just going to be measuring how fast they can all call out to PCRE.
They're neat and all and it is called the benchmarks game but I wish they'd remove the ones that end up getting gamed like that.
Benchmarking programming languages got better. E.g.: http://benchmarksgame.alioth.debian.org/ gives roughly idea about performance of programing languages.
I wish something similar existed for databases. I think exact figures would be hard to get, but I believe there are many 2x 10x differences that we should be aware of.