THE ONLY BUYING GUIDE TO AUTOMATED END-TO-END TESTING YOU’LL EVER NEED

What is automated end-to-end testing?

End-to-end testing (sometimes called regression testing or User Acceptance Testing) is a form of “blackbox” testing that goes through all the user flows of an application from beginning to end, testing software the same way a user would experience it. Whereas unit testing checks a single component of a system and integration testing checks software modules as a group, end-to-end testing checks the entire system from start to finish.

Some companies go through their application manually, either by having employees click through the experience or by outsourcing the work to low-cost testers overseas. Both approaches are time consuming, prone to errors, and pretty terrible at finding bugs.

The value of automating your end-to-end tests

  • 🔍 Find bugs earlier and ship better products
  • ⚡️ Speed up development and improve velocity
  • 💰 Save money and increase productivity
  • 🌏 Accelerate innovation and capture more of the market
  • 💜 Set higher NPS scores
  • ✌️ Expand your engineering team
  • 🔒Stay compliant with regulations and governance practices

Automated E2E tests and the importance of "shift left"

“Shift left” is the practice of moving QA and testing closer to the development phase of the software development lifecycle (SDLC) so that engineers get more rapid feedback on their code. The idea is that it’s cheaper and easier to fix bugs while the team is working on the feature than after it has shipped.

Unfortunately, many companies take that to mean developers should write and maintain end-to-end tests themselves, which actually costs teams money, productivity, and their competitive advantage.

'Shift left' chart. Graph depicting the movement of QA and testing closer to the development phase.
To the left, to the left

The most effective way to shift left — and ensure a high quality product — is with dedicated testers and high-speed testing infrastructure. Developers need rapid test results every time they merge their PRs, but it's not realistic to have them maintain E2E test coverage on their own.

There are plenty of factors to consider when determining which automated testing solution is the right match for your team. Read on for more information about how to find a solution that aligns with your product, team size, budget, and overall testing capabilities.

Investing in comprehensive automated test coverage benefits everyone.

  • Developers get near-instant feedback on their code so they can meet those end-of-sprint deadlines without waiting for the QA team to review. That gives them more time to work and more time to innovate — and even more time to relax.
  • Product managers can get more accurate timelines from their engineering partners so they can coordinate better with legal, marketing, and customer support.
  • Customer support & success will see fewer support tickets and increase response time, saving the company money and improving the overall customer experience.
  • Designers can get back to design and research without the burden of UAT and manual end-to-end testing to “green light” each release.
  • Leadership can get to market faster without the risk of outages and embarrassing bugs — meaning more users and more sales.

Step 1: Assessing your process, and QA bottlenecks

Before bringing in new tools and revamping your development process, start by reviewing your team’s actual needs and specific pain points. When you know what's missing from your current process it'll be easier to find the right QA solution for your team.

Ask yourself if any of these common problems sound familiar:

Are you releasing as often as you want to be?
If your QA process — or lack of one — is slowing down the development process, you’re wasting money and losing your competitive advantage.

Are bugs getting out?
When bugs end up in production they reduce trust, brand loyalty, retention, and finally revenue.

Are there bottlenecks in the deployment process?
This is usually caused by flaky (or missing) automated tests; complex, feature-flagged testing environments; and integrations with third-party services.

Are developers endlessly babysitting their builds?
When tests runs sequentially, instead of in parallel, the whole suite can take hours to run — time that your developers could be using and never get back.

XKCD comic. Programmer comic
xkcd.com/303

Are developers re-doing work they’ve already shipped?
Engineering time is scarce and expensive. The time spent fixing bugs after they've been released is a huge cost you may not even be accounting for.

Are QA engineers backlogged by last-minute testing?
Waiting until the last day of the sprint to start testing means that releases get delayed or features get released without enough testing.

Step 2: Determining the features and capabilities you’ll need

Every team is going to need something different from a QA process depending on the product, how the team is set up, and where the bottlenecks are. Once you have a good idea of how your process could be improved, you can start to think about the different features and capabilities that would work best for your team and make your process as efficient and effective as possible.

Features to look for in an automated testing solution

A. High levels of test coverage

We recommend maintaining at least 80% test coverage or higher. Today’s software is complex and interconnected; teams need comprehensive coverage to ensure each release is bug free and ready for production. High levels of test coverage give confidence that every user flow is tested and re-tested when developers release new code.

B. Testing capabilities

Not every testing solution will be able to handle every use case. To get comprehensive test coverage — 80% or higher — make sure that there will be support for:

  • ☁️ APIs
  • 🖼️ iFrames
  • ✉️ Emails
  • 📱 SMS
  • 🖥️ Multi-tab
  • 👯 Multi-user flows
  • 🧩 Chrome extensions
  • 🪄 Auth0/Magic links
  • 👆 Drag & Drop
  • 🤝 Third-party integrations (i.e., Salesforce)

C. Parallel test run infrastructure

Look for solutions that can run a whole test suite in parallel. Test suites that run sequentially (one test case at a time) can take hours to run, which saps productivity. But be careful: there are usually a steep fees for running more than a few tests at the same time — read the fine print.

D. Around-the-clock test maintenance

Automated tests need constant attention and maintenance. As you think about a solution for automated testing, make sure you think about how you will triage test failures, verify bugs, and fix broken tests. If this work isn’t done quickly, your developers won’t get the rapid feedback that they need and your QA process will start to break down.

E. Zero vendor-lock in/test portability

Writing tests with an open-source framework like Microsoft Playwright, or widely supported frameworks like Selenium and Cypress, make it possible to switch platforms as your needs change, and make it easier for new engineers to come in and support legacy tests. Choosing a solution that uses a proprietary framework makes it a lot more complicated and costly to adapt in the future.


Note: Many solutions talk about their ability to “export” test code, but be careful to verify those claims. Often times the exported code can only be run on that vendor’s internal platform. Look for solutions that write code and run tests within a testing framework in these solutions, the same test code is simply shared, not exported.

F. Test stability and reliability

End-to-end tests can be kind of fussy. If they’re not written with a focus on stability and reliability, they can flake — which means they fail even though nothing is actually wrong with the application. False alarms pull developers off their own work to investigate whether there’s a legitimate bug or a bad test. As you think about how to include automated testing in your development process, pay special attention to how you will handle flaky tests and what resources will be needed for test triage and maintenance.

G. CI/CD integration

Having a test suite run every time a developer tries to merge code is one of the best ways to keep bugs from sneaking into production. To do that, you’ll need a solution that can be integrated into your CI/CD pipeline, typically through GitHub actions or a Zapier integration.

Step 3: Reaching your automated test coverage goals

There's no "one way" to get high test coverage. Every team is going to need something different. In this section we'll talk about four sources of test coverage, their benefits, drawbacks, and costs:

• Building an in-house team
• Buying no code/low code tools
• Hiring hourly contractors
• Working with managed QA partners

In-house QA teams

A. High levels of test coverage
C. Parallel test run infrastructure
D. Around-the-clock test maintenance
G. CI/CD integration

An in-house QA team is typically responsible for writing test plans, building out test cases, executing tests, and reporting on bugs and other issues that are found during testing. The QA team may also set and enforce engineering standards like Test-Driven Development (TDD) or Behavior-Driven Development (BDD) to make sure that the product is easily testable. 

Done right, an in-house QA team provides an enormous amount of value to an organization. Their expertise in testing strategies and testing tools, combined with their familiarity with your product and customer can help developers ship faster and deliver a better user experience overall. 

But getting it right takes a significant investment in both people and infrastructure. Without the right systems in place, end-to-end regression quickly becomes a bottleneck, or gets side-stepped by fast-moving teams.

Expect to hire one QA engineer for roughly every three developers. Based on our experience, a team needs about 25 test cases per developer to maintain 80% end-to-end test coverage; and a full-time QA engineer can maintain about 50–100 depending on their skills and the complexity of the product. 

You'll also need to bring on SDETs to handle testing infrastructure and managers to oversee the new team.

Estimate the cost of your in-house team
Use this calculator to estimate up-front and ongoing costs of maintaining comprehensive end-to-end coverage with an in-house team.

Quick look: Building an in-house QA team

Benefits
👍 Understands your application and business goals

👍 Advocates for more testable, quality code

👍 Customize infrastructure to meet your needs

Drawbacks

😔 Expensive cost center; reduces resources for development

😔 Complicated infrastructure

No-code or low-code tools

G. CI/CD integration

As the name suggests, a no-code testing tool can create and execute automated tests without a technical expert to write any test code. Typically, there’s a user-friendly interface and a set of predefined actions and assertions that can be combined to create test scripts.

Because no-code tools are (reasonably) simple to use, non-technical members of your team like business analysts, customer support reps, product managers, and designers can create and maintain automated tests.

But the tools come with major drawbacks: If the DOM or CSS isn't predictable, the test creator will struggle with complex user flows. You probably won't be able to test third-party integrations, APIs, iFrames, or multi-tab and multi-user workflows.

You will also run into test run limits; and the tests themselves can’t be exported or used anywhere else.

Quick Look: No-code and low-code testing platforms

Benefits
👍 Fast set up
👍 Easy to use
👍 Spread the burden of QA across multiple teams

Drawbacks

😔 Can only be used in simple test cases
😔 Vendor lock-in
😔 Limited number of test runs and no parallelization

Hourly QA contractors

Typically based off-shore, these contractors will write the automated tests according to a testing plan and run them on pre-set schedules. Different contractors will have different set-ups but expect to negotiate:

• Coverage levels and technical abilities
• Hourly rates
• Maintenance SLAs
• Test run times
• Parallelization costs

Keep in mind: Even though the hourly rate is low, their incentive is to work slowly and run up their billable hours.

Estimate the cost of hourly QA contractors
Use this calculator to estimate up-front and ongoing costs of using hourly contractors.

Managed QA / Test Coverage as a Service

A. High levels of test coverage
B. Testing capabilities
C. Parallel test run infrastructure
D. Around-the-clock test maintenance
E. Zero vendor-lock in/test portability
F. Test stability and reliability
G. CI/CD integration

Test Coverage as a Service (TCaaS), or managed QA, is an emerging segment within automated end-to-end testing. TCaaS provides guaranteed levels of test coverage and the test run infrastructure, in addition to ongoing test maintenance, test triage, and bug reporting. However there are major differences between different providers and you will have to compare your choices carefully.

What to look for with TCaaS providers:

✔ Guaranteed test coverage levels
Many TCaaS providers will overstate the level of test coverage you’ll receive from their test cases. To maintain comprehensive coverage, expect to need about 25 test cases per front-end developer on your team.

✔ Test triage and maintenance commitments
Make sure you understand how the TCaaS provider will respond when tests break or start to flake; how quickly they will respond and take action; and the provider’s working hours.

✔ Test run infrastructure
If the test infrastructure only runs a few tests at a time, a small suite of just 200 test cases (at 5 minutes a piece) will take 10–15 hours to finish. Look for TCaaS providers that run the entire suite in parallel, and don't put limits on your test runs.

✔ Testing frameworks and the potential for vendor lock-in
TCaaS providers should use an open-source or widely-used testing framework that gives you ownership of the test code. If the provider isn’t working out, you want to be able to take your tests in-house or to another vendor — you don’t want to restart your test coverage from zero.

Step 4: Navigating the buying process

Even if you decide to build an in-house QA team, you'll need testing infrastructure and tools. In this section you'll learn what to expect when you talk to vendors, and what to ask about.

  1. Get a demo of their capabilities
    Have each vendor you talk to demonstrate how they would be able to automate test cases for your technology, and how they have solved testing challenges similar to yours in the past.
  2. Evaluate software with your development and QA teams
    Make sure you understand how the implementation goals, service commitments, and capabilities. Compare those to your team's needs and your development process to find the solution that works for you.
  3. Check references
    When you talk to references, ask about cost, level of effort to manage the vendor, responsiveness and customer service, and of course, the quality of the test coverage.
  4. Read case studies and testimonials
    Case studies can show you how well a testing solution solves a specific problem and provide insights into the client experience. Look for: Client testimonials, a clear understanding of the client's goals, and real impact.

Step 5: Winning over internal stakeholders

You may need to get approval from internal stakeholders before moving forward with a testing solution. Automated end-to-end test coverage benefits everyone, so make sure that technical and non-technical leaders see the value.

Engineering leadership

  • Takes the burden of blackbox testing off developers so they can focus on whitebox tests and code quality
  • Provides rapid feedback to engineers during the development process and throughout each sprint
  • Increases overall velocity by giving more time back to developers
  • Saves money that can be used to bring on additional engineers

Product leadership

  • Increases team velocity by reducing the QA process from weeks or days to minutes
  • Improves overall product quality and the user experience
  • Reduces the likelihood of rework and bug fixing in future sprints

Front-end developers

  • Identifies bugs quickly and accurately so they can focus on feature development
  • Eliminates the noise of flaky tests

QA engineers

  • Gives QA engineers time to work on testing strategy and test-driven development standards
  • Frees up resources for performance testing, load testing, linting, and other higher-level testing services

Finance

  • Reduces the time and resources needed to complete testing, which can help get products to market more quickly
  • Increases the efficiency and productivity of other teams
  • Reduces overall headcount costs

Executives

  • Improves overall customer satisfaction and loyalty; builds a stronger brand
  • Assists in decision-making by providing key insights about product performance and reliability

Step 6: Onboarding with QA Wolf

As you can see, there are lots of ways your team can build out automated end-to-end test coverage. But by far the fastest, easiest, and most cost-effective way is partnering with QA Wolf. If you decide to work with us, here’s what you can expect.

Phase 1: Requirements gathering 

Kick-off & product tour
We’ll learn about your application and testing priorities directly from your team during a walk-through and conversation with your dedicated QA lead.

Access & integrations
Next is getting access to your environments with log-in credentials and useful documentation, creating shared Slack channels. Some teams also like to integrate their issue trackers (Jira, Linear, etc.) so tickets can be created automatically from bug reports. 

Develop a test plan
The detailed suite of test cases will be prioritized to make sure the most important areas of your application are covered first. 

🎉 MILESTONE 1: Test plan sign-off
4 weeks from kick-off
Once you approve the test plan and prioritization we shift into test development. 

Phase 2: Implementation

Test development 
Your test coverage will ramp continuously until you’ve reached your coverage goals. 

Test maintenance
Your tests are monitored 24/5 to make sure they’re always performing, even as your UI changes and new features are added. 

Bug reporting
Your dedicated QA Wolves will investigate every test failure. Flaky tests are fixed and human-verified bugs get sent to your team. 

CI/CD integration
If it’s helpful, we can connect QA Wolf to your deployment pipeline so tests run as you deploy and give you instant go/no-go decisions. 

🎉 MILESTONE 2: Initial test suite complete
8–12 weeks from kick-off
But the work never really stops. We’ll maintain and update your test suite as the product changes, and create new tests as you release new features. 

Figuring out how to get comprehensive end-to-end test coverage is a confusing experience, especially if you’ve never done it from scratch. But the pain and embarrassment of bugs in production is even worse. In this guide we’ll help you determine what your team needs, and how to select a testing solution that works for you.