Software testing is an important part of software development, this post has a few points to consider when developing tests for your own software.
Read more about the profession of software testing
Existent
I wanted to put this one first. Before worrying about getting the testing automated, or comprehensive or whatever, the most important thing is just to have some level of testing. Tests will give you more confidence to build and enhance your software and that you can safely make releases.
Easy To Run
I’ve included ‘automation’ on this list, but I wanted to emphasise that automation is not the be all and end all. Having tests that are easy to run will mean that they actually are run. ‘Easy’ is a spectrum – try to find ways to make it easier to run your existing tests. This could involve some automation, but just adding some better documentation could help too.
Not Just the Happy Path
It is tempting to use testing to demonstrate that something works, without testing to see what happens in less than ideal conditions. This is sometimes referred to as testing the ‘happy path’. Try to think of ways to break your software. When writing tests you may want to consider including cases for:
- Valid data with correct outcome – the ‘happy path‘
- Exceptions and errrors
- Naughty data – such as the big list of naughty strings
- Extremely large or small values
Projects such as the can help you with breaking your software.
Automated
Automating your tests will make it much easier to run them at all and to run them regularly. Tests are only valuable if they are run.
Independent
Keeping tests independent will make it much easier to find out what is going on when there are problems or errors. If a test behaves differently depending on which tests ran before it, it is not independent. This is related to the point on ‘isolation’
Isolated
You should try as far as possible to isolate your tests and testing environment from external factors. At the level of individual unit tests this is relatively easy, where you can provide data or mocks for individual units. You may find you need to work harder to keep the whole process isolated from external dependencies.
This can be challenging for integration tests, or data-tests which require the use of a database. Isolation (and independence) can break down if there is assumed access to a particular test database or schema. Improving independence and isolation are ways that tests can be made easier to run.
Quick
The longer tests run, the less likely it is that they will actually be run. As a rule of thumb you don’t want your unit tests to take more than a few seconds to run, and ideally integration tests would run in more like a few minutes.
Reality often gets in the way of theory and it is not unheard of for integration test suites to take hours or even over night to complete.
One way to keep test run times down is to resist the temptation to add more and more ‘end to end’ tests and instead focus on quick-running, well isolated unit tests.
End to End
You should aim to test your software from start to finish. Try to avoid just testing individual components on their own (although you should do that too). It is all to easy to have a broken system made up of ‘perfectly working’ sub-systems and components. Testing in an end to end fashion like this helps flush out any issues with integration between components. That said, you should resist the temptation to add more and more ‘end to end’ tests, as they will slow down your testing to the point that tests will not be run as often.
Reproducible
It should be possible for you to run your tests today and then in a week or a month and be confident that you will get the same results. This will be made much easier if you have suitably isolated your code and environment for testing. For example if you software relies on external data or services, you should try to isolate your software from these. This also means nailing down any random seeds or implied randomness such as sort-orders etc.