If you’re not familiar with the Test Pyramid, it looks like this—
It was published by Mike Cohn in his book Succeeding with Agile and illustrates the hierarchy of automated testing. You’ve got unit tests at the bottom, integration tests built on top of them, and end-to-end (E2E) tests built on both. The size of each level represents the number of tests you’re likely to have, and often serves as a guide for how teams allocate their developer and QA resources.
➡ Learn more: White-box vs. Black-box testing
Cohn points out that the tests at the top of the Pyramid are more comprehensive and representative of the actual user experience. But as you go up the Pyramid, the tests also become harder to write and slower to run—two ways of saying more expensive for the team. Although, in our opinion, his cost calculation under valued all the other things teams get when they invest in high coverage.
Various updates over the years have tried to refine the model, but they all write off E2E testing for basically the same reasons.
Let’s take a look at why, and debunk some of the myths about E2E testing.
Like all the other myths around E2E coverage, this one was true when the Test Pyramid was published. Teams have basically had just two options: in-house or outsourced.
The cost of an in-house adds up quick. Here’s some back of the envelope math: A team with 5–8 developers needs about 200–300 E2E tests to cover all their edge cases, from what we’ve seen. While a good, full-time QA engineer can probably maintain 50–100 tests on their own (depending on how much the team is shipping). So to reach high coverage levels and stay there, you’ll need 2–4 QA engineers, and maybe a manager, maybe SDET, and infrastructure… Could be $300,000–400,000 a year.
Outsourcing by the hour is another option but you have to weigh the management burden and the questionable quality of work.
We built the QA Wolf platform on Microsoft Playwright and a scalable Kubernetes infrastructure so that we could offer a new approach to QA: Test Coverage as a Service lets us deliver 80% E2E test coverage at 1/5th the cost of an in-house team.
What makes Coverage as a Service unique is that engineering teams get everything they want—fast, reliable, high-coverage testing integrated into the CI/CD—without the overhead and infrastructure that in-house teams need.
Anything that the user could do can be automated with the QA Wolf platform. Our team of world-class QA engineers use it to write, maintain, and run your test suite on whatever schedule you need.
Test Coverage as a Service means we monitor the health and results of your test suite as soon as the first test is ready. If a test fails—when it fails—we immediately investigate whether it’s a bug or a flaky test. Flaky tests get fixed without you ever needing to be involved, and bug reports get filed within 15 minutes.
Maybe we shouldn’t call these myths so much as “past truths.”
It used to be that those 200–300 test cases we mentioned above would take a couple hours to run sequentially, one after the other, using the older systems. Even systems that can run tests in parallel can only handle a few at a time. Cypress’s public sample project, for example, takes 32 minutes to run 49 tests—so 1.5 tests at a time.
Given those delays, it's understandable that the Test Pyramid over-indexes on unit and integration tests. Waiting that long for feedback becomes a major blocker for fast-moving teams.
A Kubernetes infrastructure, like the one we use for the QA Wolf platform, solves the parallelization problem by leveraging the elasticity of the cloud. Don’t ask your humble blog post author to explain the mystic arts of K8s (if you really need to know, we’ll introduce you to our pod whisperers) the important thing is that developers get near real-time feedback on their builds and leaders get the confidence to ship bug free.
This is what most people have taken away from the Test Pyramid model—get coverage for the critical user flows, but otherwise move fast and break things. And, look, we get it. We're a start up too. But what if—and just roll with me here—we moved fast and didn’t break things?
Unit tests and integration tests are critical, but they aren’t comprehensive enough to catch all the bugs in the system. Without full E2E test coverage, teams spend a ton of time fixing what they’ve already shipped instead of moving forward.
In 2020, it’s estimated that companies spent 620 million hours of developer time debugging software. That’s an eye-watering cost of $69 billion in wasted salary (Undo, 2020).
Full E2E coverage helps teams ship faster and more confidently, because bugs get caught early before they drain a company’s time and resources. Moving fast and breaking things is a cool mantra—it’s just not necessary.
Automated test coverage has been dropping continuously. Today, two-thirds of companies have less than 50% test coverage even as developers, architects, and DevOps engineers spend more time on manual testing than any other task (Smartbear, 2021).
It doesn't have to be that way.
QA Wolf works with any web-based application, and automates any actions a user can take. With our in-house QA Engineers writing and maintaining those tests—and flagging bugs as they occur—your team can focus on building and shipping.
QA Wolf is a hybrid platform & service that helps software teams ship better software faster by taking QA completely off their plate.
© {year} QA Wolf, Inc.
