On the surface the concepts of test management are simple to grasp. One test case, run against one product, by one tester and one result recorded. Such a simple concept. Except that it’s not that simple.
When you multiply up the permutations to include different versions of testcases, different versions of the product under and even different configurations the management of these artifacts soon becomes increasingly complex.
Tracking this information becomes more and more difficult as your QA team grows, your product morphs into different editions and libraries of testcases expand. To address these complexities many teams turn to test management tools to help them keep control. However, quite often this presents its own set of issues and complexities.
Below we list 7 points that we think you should consider when developing a process or selecting a test management tool. These are points that are often overlooked when we’re being dazzled by the latest graphical reports or nicely styled web 2.0 GUIs provided by popular vendors. It’s essential to assess these points. Failure to do so can leave you spending more money on tool customization to make it fit your process. Or worse still you end up modifying your process to fit the tool. A little background investigation upfront can help you avoid these issues.
1. Releases and Configurations
You will be executing tests against different releases of your product and potentially different configurations. Does your test management tool track this information? And if it does how does it differentiate between tracking releases and configurations? For some you can add to this another layer of complexity with builds and/or iterations for releases. Furthermore you may need to track environments too. Tracking these entities can result in a complex matrix of relationships. Can your test management tool support this? If it can does it simplify this or just give you more work to manage the matrix effectively?
It’s easy to assign a testcase to a tester. It’s not so easy to manage the process of assigning large volumes of testcases to different testers where they need to run them against different releases/versions. In some cases you may need to assign at the step level, others at the testcase and even at the set level. If you need to assign at different levels then you may find you’re pushing your test management tool beyond its limits. Do you need to assign steps to different testers? Do you need to have the same group (or set) of tests assigned to different QA engineers depending on the configuration they need to be run on? Do you need to differentiate between an owner of a case and the person that the case is assigned to? No single test management tool vendor supports all of these concepts in one package. Make sure the solution you pick models the process you need to support.
Once your development team have implemented a number of fixes you need to retest those fixes. How easy is it for you to identify those tests that need to be re run? You can approach this in two different ways. Either search for testcases that failed and rerun them. Or search for defects that have been resolved and make sure you have testcases specifically for those fixed defects. Either way you need to be sure that you have a simple capability to search and pick out testcases to re-run. This is a key part of good test process. Make sure your test management tools help support you here, rather than hinder you.
4. Test Management Result Aggregation
Take the scenario where you have a core product and some variations for that product. It’s not uncommon to have a core product and then modify the product for selected customers. This usually means having a set of common product modules and then some additional modules that are either new or modified for a particular client. In terms of planning you’ll probably break these into separate projects. So you’ll have your main project where you test all of the core features. Then in your custom project you’ll have a sub set of cases that you run along with cases from the core application. However, your client with the custom release may expect a full report from your test management application. So you’ll need a tool that allows you to merge results from your core project with your custom project.
Permutations or data driven testcases are a common request for many QA teams. You need to write the testcase once then have a simple way to scale up and execute the same testcase with different sets of data. Most tools support this but it’s a model that can quickly become unmanageable when implemented poorly. So a vendor telling you that, ‘yes’, they do support data driven cases is an answer that’s only scratching the surface. Check to see how the vendor has implemented these features against real scenarios. Is this going to scale and remain manageable for your team? Or does the tool leave you spending more time managing the permutations and cases than it gives you for the execution?
6. Version Control
Version control of test cases is an important aspect for many teams (especially where they have regulatory requirements to meet). For some it’s important to know that version 2 of a testcase was executed against version 3.1 of the product. Again on the surface this seems simple. However, when you’re looking at thousands of test cases which then have 3 or 4 different versions each you’re effectively managing tens of thousands of testcase versions. It’s important to understand how your test management tool manages this (if indeed it even does). Some don’t even track this information. Some track it but don’t allow you to compare versions. Others deliver all the version control features you could ever ask for but leave you with a level of complexity which is difficult to manage. You need to find the right balance here. The balance between features and the overhead of managing, tracking and reporting on different testcase versions.
7. Master Test Libraries
Many QA teams need to develop, maintain and use a common set of cases across multiple projects. Rather like developers use common code libraries testers will use common case libraries. Again a simple concept to describe but rather more complicated to implement in practice. Complicated largely by limitations imposed by test management tools. Some tools will allow you to have common records across projects but have limited capabilities to allow updating between projects (e.g. when you update in one place, all instances are updated, even where the testcase has already been executed). Others will only allow you to copy between projects meaning it’s nigh on impossible to keep all instances updated consistently. Understanding exactly how you need to propagate these changes across your library application is key here.
The core concepts behind test management are simple. Yet the real life implementation can be incredibly complex. When we turn to tools to help us deal with these complexities it’s important to make sure you’re picking a tool that supports your process. Picking a tool on surface values will result in a solution that hinders rather than helps your team. Make sure you consider these 7 complexities before you end up with a test management solution that adds another layer of unnecessary complexity.