In an ideal world, all software would have 100% test coverage and be fully tested before anything is shipped. There would be a test case for every possible workflow and no one would have to worry about bugs sneaking into production. Unfortunately, we live in the real world where time, budget, and engineering capacity limit how much you can get done each week.
When you’re in the first stages of building out your QA tests, it’s critical to consider these limitations and make strategic decisions. Since your goal should be to prevent the most critical and visible bugs first, we recommend going broad across your whole product, before going deep on a single feature.
Let’s start with what we mean by “broad coverage” and “deep coverage.”
The product you’re testing might have hundreds of workflows, but they're not all equal. Some are highly trafficked by customers, some are highly critical for the app to function, and some generate a lot of revenue. Unlike a confirmation message on a Settings menu three-screens deep, these workflows have a huge impact on your business. When we say broad test coverage we mean looking across the entire application to ensure that all the mission-critical parts of the experience get test coverage first.
The goal with broad test coverage is to catch the most critical bugs before they are released into production. The best way to do that is to start with testing that mimics the customer’s journey through the app.
For example, imagine you run an online bank that has a web-based application. You might find that the majority of visits are for one of these five workflows:
These features are scattered across the application, and while there are hundreds of other things that your users might do, if you start with these tests you’ll be confident that new code is safe for the largest number of users.
Now, let’s consider going deep on just one of those features: Moving money between accounts. To have deep, comprehensive test coverage you’ll want to test the following workflows:
While these workflows should all get test coverage eventually, they’re not things that most users do every day. Some of these actions, a customer might never do, or only do once or twice a year. But if you build automated tests for them, you’ll have to maintain those tests at the expense of others that might be more valuable.
Maintaining end-to-end tests is a huge burden for teams, and building out tests in the wrong order can severely limit their ability to increase coverage in the future. That means more of your application is exposed and there’s a bigger risk of bugs that impact a larger set of users.
Every application will be different, but there are six main areas you can target when beginning to build out your testing plan: user activity, revenue impact, product complexity, dependencies, security, and recency.
User activity is all about where your users are spending most of their time in your app. What are the most commonly used features? Where do you see the highest amounts of traffic? The more often a workflow is used, the bigger impact a bug will have on the user experience.
Revenue impact has to do with the workflows that are critical to your business operations. This includes user registration, transactions, and key service delivery. Even minor issues in these workflows can result in significant financial loss or operational disruptions.
Product complexity refers to the parts of your product that are inherently intricate or where there are a lot of moving parts. With increased complexity comes a greater chance of bugs and ripple effects. Because there are so many things that could go wrong, it’s helpful to have automated tests that can spot problems quickly.
Dependencies are when parts of your app rely on other areas in order to function. If one part of a dependency chain is faulty, it can cause a cascade of problems down the line.
Security and compliance concerns need special attention. Failures in these areas can lead to serious legal and financial consequences. Implementing rigorous tests can ensure you meet all necessary requirements and maintain the security of your software.
Recency refers to the parts of the product that have been recently developed or modified. These areas are often less polished and potentially less reliable than code that has been thoroughly tested and refined over time. By prioritizing the testing of this new code, you can ensure that any issues are caught early before they can cause problems in the live environment.
The challenge with building and maintaining comprehensive end-to-end automated tests in-house is that you’re constantly balancing priorities, and almost always treading water without getting ahead.
Even the most senior QA engineers can only manage 50–100 test cases on their own. After that, you’ll need more people to manage more tests, and share the other responsibilities: requirements analysis, documentation, bug reporting, etc. They’ll be so focused just on maintaining the test suite they won’t have any time for the rest of QA.
That’s where QA Wolf comes in. You can have broad and deep coverage — in less than 4 months, for the cost of a single QA engineer. We free up your developers to build new features and your QA engineers to raise the quality bar across the whole organization.
QA Wolf is a hybrid platform & service that helps software teams ship better software faster by taking QA completely off their plate.
© 2020–2023 QA Wolf, Inc.
