The Corrello Blog

Thoughts on Scrum and Agile with Trello

How much automated testing is enough automated testing?

I had been a proponent of automated testing for a long time before we really adopted it at my last company. It was one of those things people wanted to do but never felt they had the time for. When we all actually started writing tests for our code was at the start of a new project, a rewrite of an existing product. The fact that on day one we could create and maintain 100% test coverage was a strong driver in writing more tests. I knew that code coverage was an imperfect metric, but it’s so easy to track it’s natural to use it. This is, I think, the key problem with automated testing that leads people to think that 100% is what they should be aiming for. When measuring coverage is so easy, and 100% coverage of your first piece of code is possible it’s natural to try and maintain that.

I think a lot of people who haven’t really considered what tests they should write also assume 100% coverage is what they should aim for. This seems to be the default opinion people come to the question with, if they even realise that there is a question to be asked.

Why 100% test coverage is actually a bad goal

I believe you should definitely not try to write “as much automation as possible” or fall into the common trap of thinking that 100% automation is an ideal to be aimed for.

Writing automated tests takes time, running them takes time and maintaining them takes time. Imagine for the sake of argument that only 50% of your tests are useful. In this case you have wasted time writing the remaining 50%. You need to wait for the other 50% to finish executing to get the results of the useful ones. And, you need to spend time maintaining them. Most of the work in software development is in the maintenance, not in originally creating the code. Less code is a laudable aim, this also applies to your tests.

So, you may well ask, how can we decide if we’ve written enough tests for a piece of code or not? Which is lucky because that’s what I’m going to tell you next :).

How to know if you’ve written enough tests

I propose splitting your tests into four groups:

  1. Core functionality of your app. If your app is a website monitoring tool then this would probably be any alerts which need to fire if a site has problems. If these go wrong your app isn’t serving its core purpose so having automated regression tests to run on every code change makes sense.
  2. The basics (often called a smoke test). For example you could use Selenium to click all the links to navigate through a web app and make sure the server responds with a 200 for each page. Note, I don’t mean checking the content of the pages, just that you can navigate the app and it’s not obviously broken. These tests ensure any manual testers aren’t having their time (and morale) wasted with obviously broken builds.
  3. Anything fragile which is likely to break. This is hard to define for new code, but if you’re working on some legacy code which you know to be bug prone then building tests around any changes you make is a good way to build towards being able to refactor more safely.
  4. Things the developers wanted to automate because it was easy to do. Sometimes it makes sense to add tests while you’re building something, that shouldn’t be discouraged.

By structuring your test cases along the above lines, when one fails people know why it was created. If it is the test that is wrong and not the code under test they should understand why it’s important to fix the test rather than delete/disable it. Or if it’s not important to maintain, they can delete it if that’s what they want to do. Note, I would suggest explicitly laying out your code using these four categories so anyone reading it knows which group the test is in.

Code review of the tests then becomes easier as someone can look at the functionality under test and form their own opinion about the tests which should appear in each category. This is where a code coverage report is useful, a scan through that as part of a code review may show something up which you think should be tested.

You can obviously adapt the above to your own needs, but I think the categories are vague enough they would suit most teams.

Agile team? Using Trello? You should check out Corrello – Dashboards for agile teams using Trello.

Previous

How to create recurring tasks in Trello

Next

Burndowns and CFDs in your Trello boards – It’s the Corrello Power-Up!

3 Comments

  1. Nice info. Thanks for sharing.
    I often like to start with this thought to make sure the testing is covered: What failures could get past your automation and testing that would make you cringe if the owner of the company came and asked you how a failure slipped by you and into production systems. If you’d cry, it should have been automated, and even as a smoke test in the dev environment, or listed as the highest priority an publicized to your boss. If you’d be nervous to explain, that means you still weren’t done in your coverage. Past that, you should be focused on maintaining tests and adding tools to assist development.

  2. Matt

    It depends what you even mean by ‘coverage’. Code? Features? Stories?

    The important part is to cover things as you go and maintain that quality, rather than approach the tests as something separate to the main development effort. IMO a story cannot be declared as truly done until it has been tested as well – preferably automatically if it’s any project at scale (there are exceptions: if it’s a once off then you probably won’t get your ROI with regards to developer time spent). You can review and refactor your tests in future, just like with code. And stories may change existing tests, just like they will force you to change existing code.

    One of the things I like about AUT is that it catches things that weren’t captured during the development effort, and you can then dedicate part of a bug’s development time to then plug that hole in the test set, if it turns out to be a major problem.

    • robinwarren

      By Coverage I was talking about automatically generated coverage metrics, so ‘code’.

      I’d agree that you should cover things as you go (as opposed to retrofitting tests after you’ve got the code ‘working’) but possibly I disagree on how much you should write. The key is that not all of those tests will have value, and in my experience some will often have negative value as they take time to execute and maintain. Thinking of tests in the groupings I present above helps them be better organised, let’s you know when a test is optional and when you have written enough tests for the code you are currently working on.

Leave a Reply

Powered by WordPress & Theme by Anders Norén