Don't write tests first. If you're not writing tests, you don't need CI to run tests, and CD isn't safe.
Write tests only:
(a) if the algorithm is so complicated, that you need unit testing to ensure that you're writing it correctly in the first place
(b) if, when, and after you earn revenue from the code
Why would you waste your time proving that something you wrote works, if you don't have product-market fit, and you need to throw it all out next week to try to satisfy the market with something different? Just make sure you have the discipline to go back and actually add edge-to-edge / integration testing after you make the sale, and not go straight to the next feature, leaving the lack of testing as technical debt.
"Why would you waste your time proving that something you wrote works, if you don't have product-market fit, and you need to throw it all out next week to try to satisfy the market with something different?"
Because somewhere around the 3-6 week mark, having decently-tested code means I move faster, not slower.
And to be honest, I'm too effing old anymore to put up with the endless loop of "oh, this change can't break anything -> ship it -> oh crap, that broke like three things -> fix them -> oh crap, that caused a regression over here -> fix it -> oh crap, that fixed that but now this other thing has regressed -> oh crap, this means my entire architecture is wrong which I would have figured out much earlier if I hadn't been so busy putting out fires". I've wondered how much people complaining about burnout is from them living in that loop. I far prefer my tests to tell me about this than my customers.
Yeah, if you're literally bashing together code in one week, to heck with tests. But I think even in the "startup" space that's a minority. Anything that can be solved in a solid month of coding is already commercially available with half-a-dozen plausible open source alternatives anyhow.
You know, I really wish that a lack of automated tests translated to poor execution and short-term business failure, but having worked for multiple successful startups where that is not the case, sadly, I was proven wrong. In the face of evidence to the contrary, I changed my opinion.
Testing needs to have business justification for it; "best practice" is not good enough. That means doing it when absolutely required for a feature's development, or tying it to revenue on the books.
I agree it might not relate to business failure but I’ll disagree on the impact to execution and I’m surprised that you’ve come to an alternate conclusion.
Maybe I’m just a really shitty coder but I’ve yet to be able to write more than ~50 lines of non trivial code bug free. To be fair though, all the research on the topic I’ve read seems to bear out that that’s the typical quality. Honestly I’m normally skeptical of such lines of research but this has typically been work that’s replicated across companies with real world data sets so while not necessarily conclusive, directionally it seems correct. With tests I can verify that all the pieces will integrate correctly. I’ve found the cost of maintaining well written tests and code to be minimal vs the time spent on everything else (eg I’m just gluing complex components together instead instead of at the same time figuring out why the overall piece isn’t working). It’s also an easy spot to add small experiments of behavior when debugging (eg “is this component behaving correctly under these conditions”) whereas if you don’t have any tests or they’re all very high level, then the cost of debugging grows rapidly.
For what it’s worth, I’ve worked on a number of complex r&d and engineering projects and have always observed the team moves more quickly with a good test suite, but I’ve also seen not good test suites that slow things down. Maybe it’s possible you’ve mostly observed bad test suites?
I highly recommend Brett Victor’s talks about what development should look like. I think the main problem with tests is that they don’t let you mutate the runtime at will / see values inline / visualize what’s happening. I think that’s the next big step that would really clinch development to be much better.
Best practice is to listen to your lawyer. Lawyers try to reduce all risk. Low risk, low reward. High risk, high reward. It's all about risk tolerance and management.
Absolutely -- Tests are a business investment, and a business expense.
A question is: "what is the business consequence if this code doesn't work as expected?" If the code is frontend stuff, then maybe tests won't pay off.
If the code is "calculate total and taxes for user buying products", then yes, you probably do want to write tests. If you don't write them first, then they might not get done until there's a "big bang" of "gee we need to write tests for the last 10k lines of code over the weekend". Or worse: code might or might not be correct in the past, but you can't tell how many users were affected.
Tests are a business investment, and it's worth a discussion to find out risk tolerance.
Nothing is more expensive than unmaintainable tests.
What I tend to want is unit tests that are cheap. Cheap to run, cheap to read, cheap to write. If they're properly simple I can rewrite them. If the requirements changed I can delete them and write new ones. If you get your testing pyramid you also know that most of your tests are cheap.
The more business logic that leaks up into your functional and integration tests, the more important it is that you are confident that the rules you have are the correct rules. Which leads into the problem space of GP.
More information is better, but more tests often give you the same information, or conflicting and therefore less information. Tests are about confidence. Can we ship this, did we break stupid things? Repeating an action for the sole purpose of soothing yourself is a form of OCD. It's not healthy.
There are a lot of things I'd love to see change in the testing world.
One would be a way to write a second implementation that is used to test or generate tests. For example if I want to test a function that adds two numbers ("add") it would be really nice to be able to write: test(add, (a, b) => a + b)
IMO tests kinda boil down to write it twice and hope you got it right once.
I routinely write a simple implementation with tests, then a fast implementation that passes the same tests, then check the two did the same thing on every call in a debug/contracts build.
Mix it with design by contact as the sibling post suggests and you get 'postcondition: did same thing as the simple implementation', which combined with tests to check said thing was broadly sensible, seems to suffice.
I worked on a project where the architect kept talking about the rewrite they were going to do next year. That was cute until it wasn't.
It wasn't cute when he started pushing back on anyone making material architecture improvements to the 'legacy' system. How do you expect people to write better code on the new system instead of the same bad code they're writing now? People don't work that way.
Shortly before my contract was up, the person who approved his rewrite got promoted out of the building. I'd bet anything that his new boss didn't give him his rewrite.
Starting a project with no culture of testing, refactoring, whatever, makes it harder to add it on later. And if you fail to get it to stick, now you're in Sunk Cost territory, because knowing everyone was going to say no early on could have let you bow out before you got invested in a project that was going to torture you.
I have a feeling you're working from a different set of base assumptions. If you're working for a company that is paying multiple people's worth of salaries to deliver a project, a project whose definition of done is "delivered to market" and not "earning $X revenue", then yeah, if people are going to be de-tasked after the project is up and migrated to other projects, I absolutely appreciate that whatever project manager is running that project should ensure that it has a nice testing suite to prove that the project works. Behavior-Driven-Development, ideally.
I just think that's a monumentally expensive way to develop software, one that executives should have relegated to the trash bin twenty years ago, after the Agile Manifesto. If you're a private sector business then you want to develop quality software at the lowest cost. Guess what? Automated tests, in and of themselves, don't tell you whether you have quality software! The only way you know whether your software has any quality at all is whether people pay money for it! Not whether software fits an arbitrary spec determined at the beginning of the "project".
It sounds to me like you're arguing from a position where tests are expensive and time consuming to write.
If you have the right habits and testing tools in place, this doesn't need to be the case.
I write tests for even my tiniest projects: I estimate I spend less than 20% of my coding time working on the tests - and they give me more than a 20% boost in productivity VERY quickly.
But... that's because every one of my projects starts with a pytest test suite in place. Adding new tests to an existing test suite is massively less work than adding tests if there's no test suite and you have to spin one up from scratch.
I’ve seen it go off the rails though. I’ve seen people exhibit traumatic responses to it.
Pain is information, but there are less helpful ways to interpret it.
There are some architectural decisions I resist energetically because they create a lot of consequences for testing, with not a lot of value added compared to other alternatives. If you introduce something that makes writing code 30% harder you’d better be getting a lot of profit margin out of the deal. Otherwise you’re just torturing people for no good reason
I haven't seen a single case where it doesn't go off the rails.
One problem with tests is that benefits are immediate, but costs are delayed. When you're writing tests it feels like you're creating nothing but value.
By the time you realize that tests made your architecture stiff as a board, they have already spread everywhere.
I've seen numerous cases where teams were not able to refactor because too many tests were failing. Quite ironic, given that tests are supposed to help refactoring.
I don’t think it’s an accident that I’ve had more testing mentors than anything else. Testing is hard. There’s a huge number of mistakes you can make and I’ve only made 70% of them so I deflect people calling me an expert. I’m not an expert, but I can look at tests and enumerate all of the rookie and sophomore mistakes, and let me tell you that can keep some people busy for their entire tenure.
I don’t say it in front of policy people much, but I immediately suspect anything as hard as this of being done incorrectly.
> It sounds to me like you're arguing from a position where tests are expensive and time consuming to write.
Of course they are. Tests are code. As a rule of thumb, sufficient coverage requires writing three times as many lines of code for the test suites than for the code it's testing. You need "sufficient coverage" to get the benefits of "I completely forgot these parts of the codebase, but I can make changes without fear of breaking something, because wherever I would need to make changes, there are tests covering it". 3:1 test to application ratio means that it's roughly four times as expensive compared to a completely untested codebase; compensating for the cost of manual testing, so perhaps three times as expensive.
If you're only writing tests for a small portion of the codebase - a complicated section of the business logic, say - see (a) in the original comment, where I do see value in writing tests up front. But that doesn't provide full coverage.
I gave a talk at DjangoCon today about how I maintain 185 projects as a solo developer... thanks to a habit of comprehensive tests and documentation for every one of them. Video online soon, slides and notes are here: https://github.com/simonw/djangocon-2022-productivity
I am confident I would not be able to productively maintain anything near this much software without those tests!
Detasked is a definite problem, but I’m more concerned about the reverse, which is being overwhelmed by customer input. Once the party starts it’s too late to change some things. Everything is a priority queue and louder more charismatic voices will push out you quality of life concerns.
Edit: but in this thread what I’m talking about is skill. The longer you wait to practice a skill that you will need in abundance, the more likelihood that you will fail outright. You need a long runway for operational excellence.
You're using a different definition of "quality software" that is orthogonal to the one that the rest of us are using. Your definition is the one that isn't helped much by automated tests. But for an actual product, you really want both kinds of quality, including the kind that is helped by automated tests.
(c) You start wasting a lot of time fixing the features you previously shipped but broke later.
When you start to waste so much time manually testing stuff, and realizing that something you implemented and worked two weeks ago, but now doesn't because someone forgot to check that their new work didn't break the old one.
Why are you keeping around features that aren't tied to revenue? Either write tests for the features, or delete the features. The easiest code to test and maintain is no code at all.
Honestly, in most cases testing is a band-aid on top of a bigger problem. IME things usually break due to complexity, not due to lack of tests.
But of course, tackling complexity is hard. It's much easier to sprinkle the big ball of mud with tests and set it in stone.
If you feel like you're wasting time manually testing, it probably means that the blast radius of your changes is too big. You should prefer smaller rockets over your head, not bigger bunkers to hide in.
Some stuff just takes time to manual test, e.g. if you provision multiple cloud resources and must coordinate them. Or anything involving some sort of multi step process. If you want to ship quickly, you want a test suite to save you the trouble of doing an hour long of manual test for each change you're adding. Delegating the verification to computer, so that my small team can move faster on the other things we need to build, is a no-brainer. At least, once the "cost to manually test" becomes greater than "cost to implement some tests and setup CI".
... especially with some of the most-popular "let's shit out a quick MVP" languages/frameworks, since they often lack static typing, or kinda have it but support is uneven.
Writing automated tests with sufficient coverage takes, as a rule of thumb, three times as many lines of code as the code itself, not including test data. (I don't remember exactly where I first learned this, I think this is out of Microsoft?).
For small-enough codebases, with few enough engineers working on those codebases, and without dedicated QA resources attempting to break your code on a thousand device permutations, simple manual checks, when needed, are orders of magnitude faster compared to writing and maintaining automated test suites for code that nobody is paying you money for yet and therefore is worse than worthless from an accounting perspective, due to ongoing maintenance costs of the code itself.
Once you have paying customers, then you need the tests to protect company revenue and reputation. Once you decide to scale, then pay for QA to try to break your code on a thousand end-user devices. But not before!
So the question is, are you going to test it manually?
Case study 1: The help functionality in the UI was broken for months before anyone noticed, because people on the team don't need the help. Integration tests were written to exercise the help functionality. Problem solved.
Case study 2: 'We' insisted on having extensive unit tests for our login logic. Getting code coverage on that functionality ended up being easier said than done, and a bunch of refactoring was necessary to make the tests not suck balls. Said refactoring introduced a number of regressions in the login logic. Every time the login functionality broke, two engineers came to tell me or my teammate that login was broken before the CI build had even completed.
If the human is willing, faster than the automation, and failures are unambiguous, what's ultimately the value of those tests? Goodhart's Law says it's less than or equal to zero.
Great case in point. If nobody noticed, can you really tie it to revenue? Did sales fail because prospects couldn't use the help? Did customers churn because they didn't want to pay for something broken? No? Then it doesn't matter that it's "broken". Unless it affects the bottom line somehow, even in the most tenuous way, then from a business/financial perspective, arguably nothing is actually "broken".
> Every time the login functionality broke, two engineers came to tell me or my teammate that login was broken before the CI build had even completed.
So, technically, somebody manually tested it, right? Ideally, the executives should have been pressuring you to do a better job and not force other teams to catch your bad work, but it's fine if it's a rare-ish occurrence.
That depends on your delivery pipeline. At the time this product was only shipping between teams on a new initiative, so all of the users were experts. But things like this cost you status once louder more influential people start noticing.
The login one I’ve seen on two separate projectst, only one was I directly involved in. What was needed were negative tests to make sure authorization or authentication don’t succeed when they shouldn’t. I won’t begrudge those tests at all, and may insist on them personally. But some parts of the code are dogfooded hourly. If you can test it quickly and cheaply, by all means do. But if the test system is slower and less reliable, if chasing a metric makes you break things that already “worked” you need to tap the brakes and think about your strategy. Not this way, or maybe not at all.
Don't write tests first. If you're not writing tests, you don't need CI to run tests, and CD isn't safe.
Write tests only:
Why would you waste your time proving that something you wrote works, if you don't have product-market fit, and you need to throw it all out next week to try to satisfy the market with something different? Just make sure you have the discipline to go back and actually add edge-to-edge / integration testing after you make the sale, and not go straight to the next feature, leaving the lack of testing as technical debt.