I appreciated this post. Separately from what they are talking about, I found this bit insightful:
// This limits the value of testing, because if you had the foresight to write a test for a particular case, then you probably had the foresight to make the code handle that case too.
I often felt this way when I saw developers feel a sense of doing good work and creating safe software because they wrote unit tests like expect add(2,2) = 4. There is basically a 1-1 correlation between cases you thought to test and that you coded for, which is really no better off in terms of unexplored scenarios.
I get that this has some incremental value in catching blatant miscoding and regressions down the road so it's helpful, it's just not getting at the main thing that will kill you.
I felt similarly about human QA back in my finance days that asked developers for a test plan. If the dev writes a test plan, it also only covers what the dev already thought about. So I asked my team to write the vaguest/highest level test plan possible (eg, "it should now be possible to trade a Singaporean bond" rather than "type the Singaporean bond ticker into the field, type the amount, type the yield, click buy or sell") - the vagueness made more room for the QA person to do something different (even things like tabbing vs clicking, or filling the fields out of sequence, or misreading the labels) than how the dev saw it, which is the whole point.
// This limits the value of testing, because if you had the foresight to write a test for a particular case, then you probably had the foresight to make the code handle that case too.
I often felt this way when I saw developers feel a sense of doing good work and creating safe software because they wrote unit tests like expect add(2,2) = 4. There is basically a 1-1 correlation between cases you thought to test and that you coded for, which is really no better off in terms of unexplored scenarios.
I get that this has some incremental value in catching blatant miscoding and regressions down the road so it's helpful, it's just not getting at the main thing that will kill you.
I felt similarly about human QA back in my finance days that asked developers for a test plan. If the dev writes a test plan, it also only covers what the dev already thought about. So I asked my team to write the vaguest/highest level test plan possible (eg, "it should now be possible to trade a Singaporean bond" rather than "type the Singaporean bond ticker into the field, type the amount, type the yield, click buy or sell") - the vagueness made more room for the QA person to do something different (even things like tabbing vs clicking, or filling the fields out of sequence, or misreading the labels) than how the dev saw it, which is the whole point.