<  back to blog

Don’t let the Test Pyramid convince you that low E2E coverage is inevitable—or acceptable

Laura Cressman
Laura Cressman
Kirk Nathanson
Kirk Nathanson
July 28, 2022
Since 2009, the Test Pyramid has given engineering teams an excuse to under-invest in end-to-end coverage. The feeling was that only a few critical areas were worth the time, effort, or expense. But new services and technologies have lowered those barriers, and the teams that take advantage of them ship faster, provide a better experience, and ultimately are more competitive.

If you’re not familiar with the Test Pyramid, it looks like this—

The Test Pyramid (h/t Mike Cohn, 2009)

It was published by Mike Cohn in his book Succeeding with Agile and illustrates the hierarchy of automated testing. You’ve got unit tests at the bottom, integration tests built on top of them, and end-to-end (E2E) tests built on both. The size of each level represents the number of tests you’re likely to have, and often serves as a guide for how teams allocate their developer and QA resources.

Cohn points out that the tests at the top of the Pyramid are more comprehensive and representative of the actual user experience. But as you go up the Pyramid, the tests also become harder to write and slower to run—two ways of saying more expensive for the team. Although, in our opinion, his cost calculation under valued all the other things teams get when they invest in high coverage.

Various updates over the years have tried to refine the model, but they all write off E2E testing for basically the same reasons. 

Let’s take a look at why, and debunk some of the myths about E2E testing.

Myth #1: Full E2E test coverage would cost a fortune 

Like all the other myths around E2E coverage, this one was true when the Test Pyramid was published. Teams have basically had just two options: in-house or outsourced.

The cost of an in-house adds up quick. Here’s some back of the envelope math: A team with 5–8 developers needs about 200–300 E2E tests to cover all their edge cases, from what we’ve seen. While a good, full-time QA engineer can probably maintain 50–100 tests on their own (depending on how much the team is shipping). So to reach high coverage levels and stay there, you’ll need 2–4 QA engineers, and maybe a manager, maybe SDET, and infrastructure… Could be $300,000–400,000 a year. 

Outsourcing by the hour is another option but you have to weigh the management burden and the questionable quality of work. 

Fact: New technologies and business models have changed the economics of E2E test coverage

We built the QA Wolf platform on Microsoft Playwright and a scalable Kubernetes infrastructure so that we could offer a new approach to QA: Test Coverage as a Service lets us deliver 80% E2E test coverage at 1/5th the cost of an in-house team.

What makes Coverage as a Service unique is that engineering teams get everything they want—fast, reliable, high-coverage testing integrated into the CI/CD—without the overhead and infrastructure that in-house teams need. 

Anything that the user could do can be automated with the QA Wolf platform. Our team of world-class QA engineers use it to write, maintain, and run your test suite on whatever schedule you need.

Test Coverage as a Service means we monitor the health and results of your test suite as soon as the first test is ready. If a test fails—when it fails—we immediately investigate whether it’s a bug or a flaky test. Flaky tests get fixed without you ever needing to be involved, and bug reports get filed within 15 minutes.

Myth #2: Full E2E test coverage would slow down the developers and deploys too much

Maybe we shouldn’t call these myths so much as “past truths.”

It used to be that those 200–300 test cases we mentioned above would take a couple hours to run sequentially, one after the other, using the older systems. Even systems that can run tests in parallel can only handle a few at a time. Cypress’s public sample project, for example, takes 32 minutes to run 49 tests—so 1.5 tests at a time. 

Given those delays, it's understandable that the Test Pyramid over-indexes on unit and integration tests. Waiting that long for feedback becomes a major blocker for fast-moving teams.

Fact: QA Wolf can run thousands of tests in parallel and give near real-time feedback

A Kubernetes infrastructure, like the one we use for the QA Wolf platform, solves the parallelization problem by leveraging the elasticity of the cloud. Don’t ask your humble blog post author to explain the mystic arts of K8s (if you really need to know, we’ll introduce you to our pod whisperers) the important thing is that developers get near real-time feedback on their builds and leaders get the confidence to ship bug free. 

Myth #3: If the happy path is covered, edge case bugs are just the cost of moving fast

This is what most people have taken away from the Test Pyramid model—get coverage for the critical user flows, but otherwise move fast and break things. And, look, we get it. We're a start up too. But what if—and just roll with me here—we moved fast and didn’t break things?

Fact: Teams that are constantly fixing bugs aren’t moving fast, they’re stuck

Unit tests and integration tests are critical, but they aren’t comprehensive enough to catch all the bugs in the system. Without full E2E test coverage, teams spend a ton of time fixing what they’ve already shipped instead of moving forward. 

In 2020, it’s estimated that companies spent 620 million hours of developer time debugging software. That’s an eye-watering cost of $69 billion in wasted salary (Undo, 2020). 

Full E2E coverage helps teams ship faster and more confidently, because bugs get caught early before they drain a company’s time and resources. Moving fast and breaking things is a cool mantra—it’s just not necessary. 

It’s time to break out of the pyramid

Automated test coverage has been dropping continuously. Today, two-thirds of companies have less than 50% test coverage even as developers, architects, and DevOps engineers spend more time on manual testing than any other task (Smartbear, 2021).

It doesn't have to be that way.

QA Wolf works with any web-based application, and automates any actions a user can take. With our in-house QA Engineers writing and maintaining those tests—and flagging bugs as they occur—your team can focus on building and shipping. 

If you’re not familiar with the Test Pyramid, it looks like this—

The Test Pyramid (h/t Mike Cohn, 2009)

It was published by Mike Cohn in his book Succeeding with Agile and illustrates the hierarchy of automated testing. You’ve got unit tests at the bottom, integration tests built on top of them, and end-to-end (E2E) tests built on both. The size of each level represents the number of tests you’re likely to have, and often serves as a guide for how teams allocate their developer and QA resources.

Cohn points out that the tests at the top of the Pyramid are more comprehensive and representative of the actual user experience. But as you go up the Pyramid, the tests also become harder to write and slower to run—two ways of saying more expensive for the team. Although, in our opinion, his cost calculation under valued all the other things teams get when they invest in high coverage.

Various updates over the years have tried to refine the model, but they all write off E2E testing for basically the same reasons. 

Let’s take a look at why, and debunk some of the myths about E2E testing.

Myth #1: Full E2E test coverage would cost a fortune 

Like all the other myths around E2E coverage, this one was true when the Test Pyramid was published. Teams have basically had just two options: in-house or outsourced.

The cost of an in-house adds up quick. Here’s some back of the envelope math: A team with 5–8 developers needs about 200–300 E2E tests to cover all their edge cases, from what we’ve seen. While a good, full-time QA engineer can probably maintain 50–100 tests on their own (depending on how much the team is shipping). So to reach high coverage levels and stay there, you’ll need 2–4 QA engineers, and maybe a manager, maybe SDET, and infrastructure… Could be $300,000–400,000 a year. 

Outsourcing by the hour is another option but you have to weigh the management burden and the questionable quality of work. 

Fact: New technologies and business models have changed the economics of E2E test coverage

We built the QA Wolf platform on Microsoft Playwright and a scalable Kubernetes infrastructure so that we could offer a new approach to QA: Test Coverage as a Service lets us deliver 80% E2E test coverage at 1/5th the cost of an in-house team.

What makes Coverage as a Service unique is that engineering teams get everything they want—fast, reliable, high-coverage testing integrated into the CI/CD—without the overhead and infrastructure that in-house teams need. 

Anything that the user could do can be automated with the QA Wolf platform. Our team of world-class QA engineers use it to write, maintain, and run your test suite on whatever schedule you need.

Test Coverage as a Service means we monitor the health and results of your test suite as soon as the first test is ready. If a test fails—when it fails—we immediately investigate whether it’s a bug or a flaky test. Flaky tests get fixed without you ever needing to be involved, and bug reports get filed within 15 minutes.

Myth #2: Full E2E test coverage would slow down the developers and deploys too much

Maybe we shouldn’t call these myths so much as “past truths.”

It used to be that those 200–300 test cases we mentioned above would take a couple hours to run sequentially, one after the other, using the older systems. Even systems that can run tests in parallel can only handle a few at a time. Cypress’s public sample project, for example, takes 32 minutes to run 49 tests—so 1.5 tests at a time. 

Given those delays, it's understandable that the Test Pyramid over-indexes on unit and integration tests. Waiting that long for feedback becomes a major blocker for fast-moving teams.

Fact: QA Wolf can run thousands of tests in parallel and give near real-time feedback

A Kubernetes infrastructure, like the one we use for the QA Wolf platform, solves the parallelization problem by leveraging the elasticity of the cloud. Don’t ask your humble blog post author to explain the mystic arts of K8s (if you really need to know, we’ll introduce you to our pod whisperers) the important thing is that developers get near real-time feedback on their builds and leaders get the confidence to ship bug free. 

Myth #3: If the happy path is covered, edge case bugs are just the cost of moving fast

This is what most people have taken away from the Test Pyramid model—get coverage for the critical user flows, but otherwise move fast and break things. And, look, we get it. We're a start up too. But what if—and just roll with me here—we moved fast and didn’t break things?

Fact: Teams that are constantly fixing bugs aren’t moving fast, they’re stuck

Unit tests and integration tests are critical, but they aren’t comprehensive enough to catch all the bugs in the system. Without full E2E test coverage, teams spend a ton of time fixing what they’ve already shipped instead of moving forward. 

In 2020, it’s estimated that companies spent 620 million hours of developer time debugging software. That’s an eye-watering cost of $69 billion in wasted salary (Undo, 2020). 

Full E2E coverage helps teams ship faster and more confidently, because bugs get caught early before they drain a company’s time and resources. Moving fast and breaking things is a cool mantra—it’s just not necessary. 

It’s time to break out of the pyramid

Automated test coverage has been dropping continuously. Today, two-thirds of companies have less than 50% test coverage even as developers, architects, and DevOps engineers spend more time on manual testing than any other task (Smartbear, 2021).

It doesn't have to be that way.

QA Wolf works with any web-based application, and automates any actions a user can take. With our in-house QA Engineers writing and maintaining those tests—and flagging bugs as they occur—your team can focus on building and shipping. 

Category