We all want high test coverage that’s stable, reliable, and fast because it gives our teams the confidence to innovate and ship faster, knowing that bugs will be caught before they sneak into production.
Our customers usually have two reasons why they tried to have developers write and maintain end-to-end (E2E) tests, before bringing on QA Wolf to help:
- They believe developers will ship fewer bugs if they’re responsible for writing and maintaining tests themselves—thereby improving overall code quality; and…
- They believe it will speed up the software development life cycle if the same people coding are the ones writing end-to-end tests.
Both of those reasons make perfect sense! After all, who knows the product better than the developers who build it?
What we find is that neither of those things really work out in practice. The teams don’t end up with the code quality, the speed, or the test coverage they want. In fact, they end up losing productivity, money, and their competitive advantage.
Let’s talk about why that happens.
Lost productivity: Developers will spend 20–40% of their time maintaining test coverage
It’s easy to think that writing a test case is a one-off thing, but the real work is maintaining the test as the product changes. We’ve seen that an experienced, full-time QA engineer can maintain 30–50 test cases on their own, but even a small team needs around 200 test cases to adequately cover their product. This is why in-house QA teams typically hire 1 or 2 full-time QA engineers for every 5 developers.
From what we’ve seen, when you pile that work onto the developers they’ll lose at least two days a week (50–100 days a year) just maintaining their test suite, which is time they’re not building and shipping.
Another major drain on developer time is waiting around for test results, or “babysitting builds.” The tests themselves can take a couple minutes each, assuming they don’t fail because of a network issue or flake out — which they will — adding up to a few hours in a suite of 200 cases.1 So your developers are stuck running, re-running, and de-bugging flakey tests when they should be building.
→ Use our QA Cost Calculator to estimate your company's needs
Lost money: Maintaining test coverage will cost thousands (maybe millions) of dollars a year
What usually happens next is that roadmaps get stretched out and features get cut for time (see the next section for more on that). But companies don’t usually think about the economics of that wasted productivity.
Engineering time is expensive. With average salaries starting around $200,000/year it’s probably the single largest cost center in your organization. So the 20–40% of the development time spent on QA is costing you $40,000–$80,000 per engineer.
That’s a huge investment which isn’t returning any new features or functionality. And doesn’t even seem to be returning high quality test coverage—90% of companies still have less than 75% test coverage (SmartBear, 2021).
Lost competitive advantage: Lost productivity will cost you market share and revenue
This might be the scariest loss of all.
With so much development time going into QA, engineering teams ship slower and fix fewer bugs. The slower pace of development and innovation is a major business risk and puts you at a significant competitive disadvantage.
And when the team starts falling behind deadline, they start to prioritize shipping over testing. Suddenly all that “developer discipline” gets sidelined, coverage levels drop, and more bugs go out to customers.
It’s not that asking developers to manage E2E testing is inherently a bad idea—the thought process makes sense—it’s just that the reality of building products and shipping fast makes it impossible for developers to do both at the same time. Despite some evidence to the contrary, developers are still only human.
There’s a better way. Test coverage as a service increases developer efficiency and output so your team can ship quickly and confidently.
So back to the top: We all want high test coverage that’s stable, reliable, and fast. The question is where that testing is going to happen and who’s going to own it.
- Do you hire an in-house QA team and build out the infrastructure? Pricey.
- Do you outsource testing and pay by the hour? Questionable quality and a pain to manage.
- Or, do you have your developers spend two days a week testing? Wasteful. And not realistic.
QA Wolf has a new approach. End-to-end test coverage as a service: 80% coverage in four months or less without a vendor lock-in, and unlimited test runs. We can do that by hiring world-class QA Engineers on staff, and equipping them with our proprietary testing platform. Together they give your developers near real-time feedback—which is the real goal of shifting left—ownership over code quality, and all the time they lost doing their own QA.
Schedule a demo to see how it works and we’ll help you compare the cost of different QA options so you can make the best choice for your business and customers.
---
1 The best case scenario for a test suite of 200 cases, each taking 1–2 minutes, and a CI capable of running 4 at a time in parallel is about 90 minutes from beginning to end. QA Wolf can run your entire test suite in parallel, bringing the time down to a few minutes—with unlimited runs at no extra cost.