How QA Wolf AI investigates a failing suite before the run finishes

Your pull request is ready, but the tests are red. A failing test exists in a sort of quantum superposition: it could be a real bug, it could be an outdated test, it could just be a problem in the environment. And until someone looks into the problem, you can't be sure your code is safe to release.

The key to a quick QA cycle and continuous delivery is being able to review and clear those failing tests as quickly as humanly — or agentically — possible.

For QA Wolf's AI, that means starting the investigation, test updates, and bug reports even while the rest of the test suite is still running.

Concurrent Execution and Investigation Slashes QA Time

In developing new agents and skills to investigate failures, we measured the real-world impact on every team that uses them. While AI evals helped us improve the agent's diagnostic abilities, at the end of the day the only stat that really matters is how much faster QA cycles have gotten.

And the best way to reduce a QA cycle is to begin triaging failures as soon as they happen, not waiting for the entire suite to run. That's what AI Investigation does: In 76% of investigations, it started before the run had finished.

We compared the same 127 recurring suites, such as a team's daily regression, before and after the rollout, and measured time to triage: from the moment every test in a suite has run once to the moment every failure in it has a conclusion. With human investigation, that's a person's diagnosis of each failure as a bug or a broken test. With AI investigation, it's the AI's last finding, such as a validated fix or a bug report. The median fell from 2.3 hours to 21 minutes, 6.5 times faster.

The gain isn't limited to the median. Before AI, 30% of suites were triaged within an hour. With AI, 72% were, and the share that took more than four hours fell from 34% to 7%. Of the 56 recurring suites with enough runs on both sides to compare, 49 got faster.

Developers and testers can hand most failing tests to AI Investigation. It settled 75% of them on its own by fixing the test, reporting the bug, or seeing it pass on a retry. When it did need someone's answer, the suite took a little longer: a median of 29 minutes, against 18.

The speed doesn't come from cutting corners. Of the tests that needed rewriting, 92% went on to pass in that run. Of the 380 bugs caught and reproduced, 317 (83%) were reported new, and 63 (17%) matched a report already open.

None of that comes from running one agent per failure. It comes from deciding, failure by failure, what the cause is and who should work on it, then making sure every fix holds up and asking a person when only a person can answer. Here's how our agentic system does that.

Agents Triage Failures Before the Suite Finishes

Because some tests are longer than others, and some start after other tests finish, a test suite doesn't report failures all at once. The suite fails gradually as the runners make their way through each flow, and waiting for 200+ tests to report their result wastes time. Instead, AI Investigation jumps in the moment a test fails.

That creates a coordination problem: A later failure may share the same root cause as a previous failure already being addressed, or a fix might expose a second problem further into the suite.

Say a redesign moves the checkout button, and the shared page object every checkout test uses still looks for it in the old place. Five checkout tests fail. The AI records one finding, "the checkout page object can't find the new button," and lists all five tests against it. When a sixth checkout test fails ten minutes later for the same reason, it joins that finding instead of starting a new investigation.

Sending each failure to its own agent would repeat the same investigation several times, and several agents might try to change the same helper at once.

So the first agent on the scene becomes the lead investigator, and every later failure goes through it, the way an on-call engineer triages support requests. It groups failures by their failing step and error, and decides one thing: Does this failure share a cause with work already under way, or is it new? A known cause goes to the investigator already working on it. A new cause gets a new investigator.

If only one test fails, the lead may fix it itself. If more fail in the same window, it delegates them to sub-agents and stays shallow, since it may be coordinating as many as 50 investigators.

An investigator stays up after its first fix, ready for more failures with the same cause. If none arrive, it shuts down, and the lead can wake it again if needed.

But fixing one problem can uncover another. In the checkout example, once the page object finds the new button, the tests run again. If one of them now gets past checkout and fails at payment, a second problem was hiding behind the first. That's treated as a new failure, with its own finding.

Grouping can shift the same way. Two tests might fail with the same timeout on the login page, one because an API slowed down and the other because its test account expired. Same error, different causes, so the work splits.

Investigators Do the Deep Work

Whether it's the lead fixing a single failure or a sub-agent handed a group, each investigator owns one cause and every test it broke. Its job is to get each of those tests to an outcome: fixed and passing again, reported as a bug, or waiting on your answer.

It works in its own sandbox, browser, and branch: the same computers we described here. It can inspect the app, edit the test code, and run the affected tests. The lead coordinates sub-agents through the QA Wolf CLI, handing each one the failure evidence and the rules for validating its work.

With that many investigators working at once, you can't watch each one. So each one works the same way: It diagnoses the failure, then follows a rule for whatever it finds.

Tell a Bug From a Broken Test

First it works out what kind of failure it has: a product bug, a test that's out of date, or a flake. A flake that passes when it runs again needs no change. A failed assertion shows what the test expected and what happened, but rarely why.

So the investigator reads the test suite's Git history, the current and previous Playwright traces, and earlier failures of the same test. When a pull request triggered the run, it reads the pull request too.

That context helps it tell an intended product change from a regression. In the checkout example, the pull request that moved the button is what tells the investigator the test is out of date, not the app. Without a pull request, the suite's history and past runs still give it evidence.

It also checks other environments. If the same test failed there and a published fix made it pass, that fix can inform the repair here.

The aim is the evidence an engineer would want before changing a test. A plausible explanation isn't enough to justify the change.

A Fix Has to Run

An agent can turn almost any test green if it's allowed to change what the test expects. So an investigator can't weaken or delete an assertion, change an expected value to match whatever the app shows now, catch an error to silence a failure, or add a fixed waitForTimeout and call it a repair.

Suppose a test expects a $90 total and the app shows $100. Changing the assertion to $100 would bury the question. The investigator needs evidence that the price was meant to change, or a person's decision.

Once it has a repair, it runs the changed code before publishing it. Until then, the edit is only a proposal.

The test also has to pass before the investigation marks it fixed, either in a full run from the investigator's browser or in a new attempt in the original run. Either way, every fix is tied to a result you can inspect, because an investigator's confidence doesn't count as a passing test.

When Intent Is Unclear, Ask

Some failures turn on product intent. If the app now applies a discount to only the first item in a cart, the investigator can show that the behavior changed, but not whether the team meant to change it.

When it can't tell, it asks you in the investigation chat, with the expected behavior, the observed behavior, and the evidence:

The test expects the discount to apply to every item. The app applies it only to the first item. Is this change expected?

Your answer tells it whether to update the test or report a regression.

An unanswered question doesn't hold up unrelated failures. The other investigators keep working, and their findings appear as they finish.

Report Each Bug Once, With Proof

When the answer is a bug, the investigator has to prove it. A failed test alone isn't enough. It has to reproduce a product defect. It can file an obvious, reproduced bug without asking, and asks first when the behavior needs your judgment.

Then it checks whether the bug is already known, because one bug can break several tests and come back tomorrow. The investigation looks at open reports, including ones linked to other tests, and at resolved reports in case the bug has recurred. When an open report covers the same cause, the investigator links the affected tests to it. When a recently resolved bug returns, its report can reopen.

It's the same rule as findings, applied to bug reports: one report per bug, not one per failing test. The match is on behavior, not the report's title. Similar wording doesn't prove two failures share a cause.

When it does need a new report, the investigator reads the team's recent reports for their format and required fields. It includes the reproduction steps, the expected and actual results, and the evidence a developer needs.

The report also comes with a video of the reproduction, with voiceover, so you can watch the failure happen and hear what's going on. Whoever picks up the bug doesn't have to reconstruct it from the investigation's conversation.

Follow the Investigation

The run page shows each finding and the tests it affects: what the investigators fixed, what they reported, and what's waiting on your answer. If you disagree with a conclusion, you can dispute it.

You can follow the same investigation from ChatGPT or Claude through QA Wolf MCP.

You can also shape how investigators work. Skills in a workspace's .skills/ folder give them instructions specific to a team's suite and environment, and teams that manage their own suites can add their own.

When you come back to that red pull request, the evidence is waiting, along with any questions only you can answer. Whether the change ships is still your call.