End to end tests posing as unit tests

I’ve been reading the excellent book The Art of Agile Development, I started to think more about why Test Driven Development is important and how it affects how you write code.

Using some of the 20%/practice group time we have at work, I started to review my own code to see if I’m living up to the test-driven development methodology. Are there enough unit tests? Yes, there were, code coverage was over 70% in most cases and with a little work I got it to 100% coverage for the most important code (the views and models). Then I started to noticed something about my tests…they’re not unit tests at all! They’re end to end tests! What Django calls unit testing is actually an end to end test. We’re testing from one end, the website, to the other end, the database. This means that we’re testing each view to ensure that it includes the right data, that it uses the right templates, etc. Then we’re testing interaction and checking the results in the database.

Example: Testing Form Validation

For example, if there’s a contest entry submission form, we will test the view that displays the form, then we check the form submission (constructing the form POST data is such a pain) and we make sure something happens with the database and that all form validators work. This is an end to end test…or maybe an integration test at the least. In no way are these small unit tests, there is a lot of setup that happens to execute the tests.

I’ve noticed that by writing end to end tests like these, I write my code to match that. Instead of creating a separate function to validate a phone number and testing that small unit, I embed that logic within the form submission or as part of a clean/save method in the model. To fully test the validation of a phone number, I will construct the form, fill it with data and then verify that the validation works. What I should be doing is testing only the phone number validation.

Let me show you what I mean with code…

If we know that the form is using the is_phone_valid function, we can assume that it will work correctly when an invalid phone number is encountered. To test the three examples used in ValidatorTests within FormTests, I would have to change the POST data of the form and resubmit multiple times. Again, this would be an end to end test because it is submitting data to a URL and checking to see if the database was modified.

Now, if I wanted to re-use the is_phone_valid function anywhere else, I would know that this single unit of code is well-tested. If there is a potential for conflict with other validation functions, I would create integration tests that use the is_phone_validfunction and combine its usage with other functions. Finally, if I wanted to ensure that every step from when a user submits the form to when a page is delivered back to them is 100% correct, then I would write an end to end test (which we can see in theFormTests class above).

Because of the way we write tests, as end-to-end tests, I wouldn’t write anis_phone_valid function, it would just be logic embedded in the form’s validation method. The end-to-end test leads me to write code that can only be tested properly with an end to end test. Writing unit tests, means you write code in smaller units that can be tested with unit tests, integration tests and end-to-end tests.

I won’t place any blame on Django for making it easier than usual to write end-to-end tests, it might just be that our company coding guidelines need to emphasize the need for smaller testable units of code and emphasize more code reviews and possibly pair programming to ensure that units being tested are small. Code reviews would also ensure that any end-to-end tests that exist have to be explained and justified.