>Good. Except, next time you hire somebody, you may get accidents.

That goes for any project in any language. You simply have to mitigate these things in a way that makes sense for your team.

>Oh, certainly, but it kind of invalidates your claim that REPL + integration tests is enough to maintain a codebase.

My point was that REPL allows you to test things quickly and can be used for many scenarios that unit tests are often used. For example, when you refactor code you can test it on the spot. The feedback loop is much tighter when you work with a REPL.

>- integration with external tools (external database, LDAP...)

The goal is to test how your code behaves and not what the external tools are doing. In my testing I capture the expected response from the external system and replay it in the test. My interest is in what the application is doing with a given input.

>- amount of setup in your application necessary prior to running scenarios

I'm not sure what this setup is exactly.

>Well, considering that a well-tested unit of code (ie, what you put in a single file) is usually dwarfed in size by the test code itself, because you're testing a lot of scenarios, then you either spend an unholy amount of time in the REPL, or you don't test everything.

The point is that the integration tests are used to test correctness. As long as they pass your application is doing what it's supposed to be doing. The REPL allows you to make sure that the change you just made does what you think it does.

>Which brings me to my second point: unit tests will run the same tests every single time. And even better, everybody in the team will run the same unit tests every single time.

Unit tests are also a huge overhead in maintenance. Any time you make a change in code, you have to go and update many tests. As you yourself point out, since you're testing many scenarios your tests often have more code than your actual application. With integration tests, you only need to change your tests when the business logic changes.

>A REPL is no substitute for unit tests. I'm not saying it's not useful, because it allows you to validate an idea fairly quickly, especially in simple cases, but it is absolutely not suited to validate the correctness of a piece of code.

I don't think unit tests are a valid way to ensure correctness either. Your individual pieces of code might be correct, but how they interact might be wrong. The only way to capture the actual use cases is by running end to end tests.

Conversely, as long as the end to end tests are passing that means that your application business logic works correctly. This is the only thing that's relevant from the user perspective.

>I fully agree with that (though running unit tests on a single unit of code is very quick).

This of course requires you to write and maintain the test for each single unit of code.

>Sure. But in my experience: most codebases have little to no integration tests, and they cover things like "make sure login is not broken" or "make sure saving an entry works" as opposed to "make sure login does not break with the login 'null'" or "make sure we reject an entry which contains invalid XML", because the cost of setting up scenarios is much higher than with integration tests.

Your argument here isn't that integration tests don't work, but teams you worked with haven't bothered using them correctly.

The same problem often happens with unit testing. I've seen many projects where people start out writing lots of unit tests, and then after a month or so things start sliding, then you have large swaths of code without tests and lots of failing tests, and then you just turn them off all together. That's a very common scenario in my experience.

> The goal is to test how your code behaves and not what the external tools are doing. In my testing I capture the expected response from the external system and replay it in the test. My interest is in what the application is doing with a given input.

Fair enough. But according to your definition, this still does not test correctness, since if you feed your actual SGBD/LDAP/etc with invalid data, your program will still produce an error.

> I'm not sure what this setup is exactly.

Consider that you have, say, a web application. You have a form, and you change the email normalization rules.

With integration tests, you simulate a POST request with a the form data (all 56 fields of them), you capture the Location header in the result, because you need to know what ID you have generated, and you do a SELECT in your in-memory database to find out if your user email is correct.

With a unit test, you simply build a user, apply the normalization function, and validate that the email looks as expected. Which is one of these scenarios is likely to be less code?

> Unit tests are also a huge overhead in maintenance. Any time you make a change in code, you have to go and update many tests. As you yourself point out, since you're testing many scenarios your tests often have more code than your actual application.

How do integration tests help with that? You're going to either end up with the same number of tests (but with more code) or you won't code the tests, and you may end up with a new issue in the bugtracker next time you release.

> I don't think unit tests are a valid way to ensure correctness either. Your individual pieces of code might be correct, but how they interact might be wrong. The only way to capture the actual use cases is by running end to end tests.

I'm actually not advocating unit tests as an alternative to integration tests, but as something complementary. It is an effective way to test the effectiveness of a unit of code, though.