Research & technology

How QA Wolf’s parallel testing infrastructure works

A well-designed test runs independently, without relying on execution order. If your tests are written to run in isolation, with no shared state and nothing left over from the tests before them, there’s no reason not to run them all at once. The hard part is the infrastructure. It has to boot hundreds or thousands of isolated test environments at the same moment, then shut them down just as fast, without bottlenecks or wasted resources.

That’s the system we’ve built at QA Wolf. When your tests run, the entire suite executes in parallel and completes in the time it takes your longest-running test to finish, whether you have 200 tests or 20,000.

Every test at once, done in the time of the longest

Compare that with a fixed number of parallel nodes. If you’re paying for 10 parallel nodes and you have 200 tests that take 5–10 minutes each, your test run could take 1 hour 40 minutes to 3 hours 20 minutes, since each node runs 20 tests one after another. With full parallelization, the same suite is done in 10 minutes, the time of its longest test.

QA Wolf

Every test at once, each in its own container

10 parallel nodes

10 tests at a time, 20 on each node

200 tests that take 5 to 10 minutes each, run two ways. QA Wolf runs every test at once, each in its own container, so the suite is done in 10 minutes, the time of its longest test. On 10 parallel nodes, each node runs 20 of the tests one after another, so the suite takes about 2 and a half hours.

200 tests that take 5 to 10 minutes each. Run all at once, the suite is done when the longest test is, at 10 minutes. On 10 parallel nodes, each node runs 20 of them one after another, and the suite takes about 2½ hours. After the first 10 minutes, the nodes' clock fast-forwards. Each test's time is illustrative.

How a run starts

Each test is stored independently in our database. A run starts when your app deploys, on a schedule, or by hand in the platform, and the system logs it as a new request with a unique build number. This number ties together the run’s results, logs, videos, metadata, etc.

We then generate what we call a run data file, an instruction set with the test code, any helper code it uses, and any configuration or custom settings the customer defined in the platform. That file goes to our runner, which allocates cloud compute resources to run the test.

A container for every test

A build of 10 tests arrives at a cluster with 3 pre-booted nodes, each running 2 pods of one container. The first 6 tests go straight into the 6 ready containers. The cluster boots 2 more nodes for the other 4, so every test gets its own container and none waits for another to finish. When the last test is done, those containers shut down, the 2 new nodes go, and the 3 nodes boot fresh containers for the next build. The next build, of 5 tests, fits in the containers that are ready.

Every test gets its own container. When a build needs more than is ready, the cluster boots more nodes rather than make tests wait. The builds, their sizes and the timings are illustrative.

Our infrastructure is built on Kubernetes clusters configured with pre-booted nodes, compute instances that are already online and ready for work. Each node runs a pre-built container image optimized for running Playwright and Appium end-to-end tests. These containers come preloaded with all required dependencies, so they can start right away once the test code is mounted.

The number of nodes follows expected demand, but test volume isn’t always predictable. When a run needs more than the capacity on hand, the system spins up new nodes on the fly, even if it has to pull capacity from multiple clusters, so every test in the suite gets its own container. As the graphic above shows, each node hosts two pods, with one container in each. Meanwhile, the pool of pre-booted containers is refilled for upcoming runs.

Once the system knows how many nodes are required, it reserves them. Each container receives the code for a single test, which we mount to its file system. Containers begin running tests as soon as they’re ready, which usually happens within milliseconds across the entire suite.

Running the tests

Tests at QA Wolf run in headed browsers, not headless mode, so we can capture video artifacts for every run. These recordings give our customers clear visibility into failures and make troubleshooting faster. Running headed adds compute overhead, but the debugging value outweighs the cost. Videos are stored alongside logs, HAR files, and other artifacts, and linked to each test’s run history in the platform.

If a test fails, we retry it up to three times before logging it as an actual failure, to rule out outside causes like an app restart or a temporary environment issue. Meanwhile, QA Wolf’s AI starts investigating failures while the rest of the suite is still running. It works out whether each one is a product bug, an out-of-date test, or a flake, then fixes the test, reports the bug, or asks you when it needs your answer.

We run each test in its own container because shared environments introduce variability, and resource contention is a major cause of flaky tests. Test execution is processor-bound, and most of that CPU goes to polling the app under test for its state. When the app is slow to respond, polling increases, and so does load. We can’t control the customer’s app, but we can isolate test execution.

Collecting results and cleaning up

Once all tests (and any retries) have finished, the system collects the results into one database record. Then we shut down all the containers and release their resources. Kubernetes handles cleanup, so no idle cloud instance is left running. That keeps operating costs low, for us and for our customers.

Getting your results

A run's results in the QA Wolf platform. A flow list: the run summary with 4 questions, status tabs (5 failed, 3 bugs, 3 blocked, 139 passed, 150 in all) and 5 failed reservation flows. Beside it, the run's results menu: 150 flows started, 5 failed and waiting on you, 3 bugs found, 3 blocked, 139 passed with 3 auto-maintained.

A run's results in the QA Wolf platform, built on our site from the product's own code.

When a run finishes, QA Wolf sends the results where you choose: email, Slack or Microsoft Teams, dashboards such as Grafana or Tableau, Xray or Zephyr in Jira, or an endpoint you host with a run events webhook, to page your on-call or start your own automation. Your deploy pipeline can also wait on the verdict before it releases. The full results are always in the QA Wolf platform.

Parallel by default, at no added cost

Traditional providers charge by the node or shard, so running more tests at once costs more. QA Wolf doesn’t, so every test in your suite runs at once by default, every time, on its infrastructure for web, mobile and desktop apps, and your runs finish in minutes, not hours.

More from the blog

All posts