That's a fact. But there are languages you are more screwed in than others. Even a tiny amount of refactoring in a dynamically typed language makes it easy to break your build, while it's much less risky in a statically typed language. And if your hypothetical team has this codebase without unit tests, we can likely assume that other best practices, such as the notion of separation of concerns, is also considered a waste of time. If you can start refactoring the existing codebase with a more modular design AND be (relatively) confident that you're not breaking your app, you'll make it possible to introduce unit tests.
The way Clojure mitigates this is by having a REPL. Whenever I work on any code I can always run it immediately in the context of the application.
Let's say I'm refactoring a function, I can run it to see what it actually does and I can write a new function and test it immediately on the spot to make sure it has the same behavior.
When you're working in a REPL you always have confidence that whatever change you made does what you want immediately. There's no recompile cycle, you don't have to restart your application and build up the state to see what effect the change has.
This allows you to refactor things and be very confident that you're not breaking anything. Coupled with high level integration tests you can be confident that all the use cases work correctly without having to have a ton of unit tests.
Having worked with Clojure it really shocks me that most modern languages do not have any REPL integrating in the editor and you still have the code/compile/run cycles.
Let's say you change your function by adding a new parameter, or you modify the order/type of existing parameters. How does your REPL save you, then? You can make sure that your new function works correctly, but not that all call sites are correct. In my experience, not that many codebases have "high-level integration tests", which are kind of an orthogonal issue compared to unit tests (you still need unit tests if you want maintainable Haskell, in spite of the type system and REPL).
The REPL lets you test any call sites immediately after you make a change. I do this all the time at work. Let's say I change a function that accesses the database, I can update the function then update the call sites and run them right there and then.
At least in Clojure, the REPL is integrated in the IDE. I can use the IDE to tell me where the call sites are what references the function, jump to definition, etc. Then I can send code from the editor directly to the REPL to test it out.
I use Intellij with the Cursive (http://cursiveclojure.com/) plugin, but you can do the same thing with Emacs and Eclipse.
My experience is that unit tests make sense in some scenarios, but not in every situation. On the other hand, high level integration tests act as a contract for your application. You should have these regardless of whether you're working with a dynamic or a static language.
> The REPL lets you test any call sites immediately after you make a change. I do this all the time at work. Let's say I change a function that accesses the database, I can update the function then update the call sites and run them right there and then.
At least the ones you know about. Now, take your codebase to 100k lines, 75% of which you didn't write. Good luck.
Wrt to integration tests: they're a pain to set up, and the complexity creeps it really easily. It's straightforward to unit-test edge cases, but the combination of layers makes it much more difficult to rely on integration tests for this kind of thing. Plus, integration tests usually take a long time to run. Rapid feedback is important.
The code base I'm working on has been developed for about 3 years by a team of 4 people. I have no trouble navigating it, nor does anybody else on the team.
If you're working with a half decent IDE, it can do things like tracking down function usages for you very easily.
>Wrt to integration tests: they're a pain to set up, and the complexity creeps it really easily.
That has not been my experience at all. Clojure has very nice testing frameworks that makes this trivial.
>It's straightforward to unit-test edge cases, but the combination of layers makes it much more difficult to rely on integration tests for this kind of thing.
The REPL allows doing a lot of what unit tests are normally used for. When you make a change you test it live in the REPL. This is no different from running a unit test.
>Plus, integration tests usually take a long time to run. Rapid feedback is important.
Hence why you don't run them all the time. The REPL provides you with rapid feedback, much more rapid than running unit tests I might add. Meanwhile, the integration tests are used for overall sanity testing.
> The code base I'm working on has been developed for about 3 years by a team of 4 people. I have no trouble navigating it, nor does anybody else on the team.
Now try that on the code base developed by 100 developers on three countries across two years.
This is what most Fortune 500 enterprise projects look like.
I have never seen any project developed by a 100 developers without it being split up into manageable chunks. This is true even for companies like Google.
You might have an umbrella project that's developed by 100s of developers. If you actually look at how the work is split up, you'll see that individual teams rarely exceed 10 people.
The joys of spending one to two months doing integrations until everything finally works, for each milestone.
Just because a few companies happen to do it right, doesn't mean the majority does it.
In one of such projects, the developers were deploying code to a live server, after producing the corresponding artifacts on their own computers. Management did not care, because it worked most of the time anyway.
> The code base I'm working on has been developed for about 3 years by a team of 4 people. I have no trouble navigating it, nor does anybody else on the team.
Good. Except, next time you hire somebody, you may get accidents.
> If you're working with a half decent IDE, it can do things like tracking down function usages for you very easily.
Oh, certainly, but it kind of invalidates your claim that REPL + integration tests is enough to maintain a codebase.
> That has not been my experience at all. Clojure has very nice testing frameworks that makes this trivial.
Hm. I've worked with "very nice testing frameworks", but I'm not aware of any which helps with the two main issues that crop up when you create integration tests:
- integration with external tools (external database, LDAP...)
- amount of setup in your application necessary prior to running scenarios
> The REPL allows doing a lot of what unit tests are normally used for. When you make a change you test it live in the REPL. This is no different from running a unit test.
Well, considering that a well-tested unit of code (ie, what you put in a single file) is usually dwarfed in size by the test code itself, because you're testing a lot of scenarios, then you either spend an unholy amount of time in the REPL, or you don't test everything. Which brings me to my second point: unit tests will run the same tests every single time. And even better, everybody in the team will run the same unit tests every single time.
A REPL is no substitute for unit tests. I'm not saying it's not useful, because it allows you to validate an idea fairly quickly, especially in simple cases, but it is absolutely not suited to validate the correctness of a piece of code.
> Hence why you don't run them all the time. The REPL provides you with rapid feedback, much more rapid than running unit tests I might add.
I fully agree with that (though running unit tests on a single unit of code is very quick).
> Meanwhile, the integration tests are used for overall sanity testing.
Sure. But in my experience: most codebases have little to no integration tests, and they cover things like "make sure login is not broken" or "make sure saving an entry works" as opposed to "make sure login does not break with the login 'null'" or "make sure we reject an entry which contains invalid XML", because the cost of setting up scenarios is much higher than with integration tests.
>Good. Except, next time you hire somebody, you may get accidents.
That goes for any project in any language. You simply have to mitigate these things in a way that makes sense for your team.
>Oh, certainly, but it kind of invalidates your claim that REPL + integration tests is enough to maintain a codebase.
My point was that REPL allows you to test things quickly and can be used for many scenarios that unit tests are often used. For example, when you refactor code you can test it on the spot. The feedback loop is much tighter when you work with a REPL.
>- integration with external tools (external database, LDAP...)
The goal is to test how your code behaves and not what the external tools are doing. In my testing I capture the expected response from the external system and replay it in the test. My interest is in what the application is doing with a given input.
>- amount of setup in your application necessary prior to running scenarios
I'm not sure what this setup is exactly.
>Well, considering that a well-tested unit of code (ie, what you put in a single file) is usually dwarfed in size by the test code itself, because you're testing a lot of scenarios, then you either spend an unholy amount of time in the REPL, or you don't test everything.
The point is that the integration tests are used to test correctness. As long as they pass your application is doing what it's supposed to be doing. The REPL allows you to make sure that the change you just made does what you think it does.
>Which brings me to my second point: unit tests will run the same tests every single time. And even better, everybody in the team will run the same unit tests every single time.
Unit tests are also a huge overhead in maintenance. Any time you make a change in code, you have to go and update many tests. As you yourself point out, since you're testing many scenarios your tests often have more code than your actual application. With integration tests, you only need to change your tests when the business logic changes.
>A REPL is no substitute for unit tests. I'm not saying it's not useful, because it allows you to validate an idea fairly quickly, especially in simple cases, but it is absolutely not suited to validate the correctness of a piece of code.
I don't think unit tests are a valid way to ensure correctness either. Your individual pieces of code might be correct, but how they interact might be wrong. The only way to capture the actual use cases is by running end to end tests.
Conversely, as long as the end to end tests are passing that means that your application business logic works correctly. This is the only thing that's relevant from the user perspective.
>I fully agree with that (though running unit tests on a single unit of code is very quick).
This of course requires you to write and maintain the test for each single unit of code.
>Sure. But in my experience: most codebases have little to no integration tests, and they cover things like "make sure login is not broken" or "make sure saving an entry works" as opposed to "make sure login does not break with the login 'null'" or "make sure we reject an entry which contains invalid XML", because the cost of setting up scenarios is much higher than with integration tests.
Your argument here isn't that integration tests don't work, but teams you worked with haven't bothered using them correctly.
The same problem often happens with unit testing. I've seen many projects where people start out writing lots of unit tests, and then after a month or so things start sliding, then you have large swaths of code without tests and lots of failing tests, and then you just turn them off all together. That's a very common scenario in my experience.
> The goal is to test how your code behaves and not what the external tools are doing. In my testing I capture the expected response from the external system and replay it in the test. My interest is in what the application is doing with a given input.
Fair enough. But according to your definition, this still does not test correctness, since if you feed your actual SGBD/LDAP/etc with invalid data, your program will still produce an error.
> I'm not sure what this setup is exactly.
Consider that you have, say, a web application. You have a form, and you change the email normalization rules.
With integration tests, you simulate a POST request with a the form data (all 56 fields of them), you capture the Location header in the result, because you need to know what ID you have generated, and you do a SELECT in your in-memory database to find out if your user email is correct.
With a unit test, you simply build a user, apply the normalization function, and validate that the email looks as expected. Which is one of these scenarios is likely to be less code?
> Unit tests are also a huge overhead in maintenance. Any time you make a change in code, you have to go and update many tests. As you yourself point out, since you're testing many scenarios your tests often have more code than your actual application.
How do integration tests help with that? You're going to either end up with the same number of tests (but with more code) or you won't code the tests, and you may end up with a new issue in the bugtracker next time you release.
> I don't think unit tests are a valid way to ensure correctness either. Your individual pieces of code might be correct, but how they interact might be wrong. The only way to capture the actual use cases is by running end to end tests.
I'm actually not advocating unit tests as an alternative to integration tests, but as something complementary. It is an effective way to test the effectiveness of a unit of code, though.
Let's say I have a function that pulls some data from the database and transforms it in some way.
I can run the original function and capture its output. Then I can write an updated function, see what its output is, I can compare them visually to ensure the change I made works as intended.
When you don't have a REPL the only way to do this is by writing a test and running it once you've made the change.
In fact, you can use the code from the tests you write in the REPL to create a unit test for later. The difference is that the REPL feedback loop is much tighter.
I didn't say it was a substitute for testing. My point was that it lets you know what the code is doing immediately. This doesn't remove the need for tests, but it means that you don't have to run tests every time you make a change.
Partially, but at least with strong typed languages, if the thing builds, it might actually run.
With dynamic ones, it is always a guessing game.
Been on this game for quite some years now, the corporate world gives zero value to unit tests, regardless of what gets discussed and presented as success stories on conferences.
