Research & technology

QA Wolf vs Claude Code vs Codex

Writing a Playwright script is routine work for a coding agent now. Claude Code and Codex can open a browser through Playwright MCP and produce a test that passes. That alone doesn't give a team a QA process it can rely on.

For that, an agent has to explore an app and work out which test cases are worth having. It needs infrastructure for the cases a browser alone can't cover, such as an email inbox that can receive a sign-in code. It has to follow the practices that keep an end-to-end suite reliable: waiting for what the app shows, cleaning up what its tests create, keeping credentials out of the code. And when a test fails, it has to investigate, so broken tests get fixed, real bugs get reported, and neither is mistaken for the other.

Those are the things our evals measure. We grade QA Wolf's own agent against them, and for this report we gave the same evals to Claude Code and Codex, on web apps and on iOS and Android apps, forgiving everything specific to QA Wolf.

All platforms · 1,567 evals

  • QA Wolfgpt-6.1-sol93%
  • Claude CodeOpus 5.578%−15 points behind QA Wolf
  • Codexgpt-6.1-sol74%−19 points behind QA Wolf
The share of evals, across web, iOS, and Android apps, where a reviewer with no QA Wolf background would accept the agent's work. Claude Code and Codex worked in the browser through Playwright MCP and on the phones through Appium.

Web Evals

The web evals cover the whole job of planning a test strategy, writing tests, investigating failures, repairing broken tests, and filing bug reports.

Web · 1,336 evals

  • QA Wolfgpt-6.1-sol92%
  • Claude CodeOpus 5.577%−15 points behind QA Wolf
  • Codexgpt-6.1-sol72%−20 points behind QA Wolf
The share of web evals where a reviewer with no QA Wolf background would accept the agent's work. Claude Code and Codex worked in the browser through Playwright MCP.

Example evals

EvalGrading
Planning

Prompt: Import the flows from this PDF. The first column has group names, the second column has flow names. Create a flow for each row.

Setup: A PDF listing 16 flows in 6 groups.

  • Creates one flow for each row of the PDF, 16 in all.
  • Groups the flows into the PDF's 6 groups.
Planning

Prompt: Find more flows. Generate 50 to 60 flows.

  • Outlines at least 45 flows.
  • Covers whole journeys, like checkout, and business logic, like pricing.
  • Leaves out flows that only navigate or take a single action.
Creation

Prompt: create a test for maps.google.com that searches for LAX and then verifies driving direction between Seattle Airport and LAX is between 10-30 hours

  • Sets LAX and Seattle Airport as the two ends of the route.
  • Asserts the driving time is between 10 and 30 hours.
  • Runs and passes.
Creation

Prompt: Write a for loop that would tell me which row number the string "TravelCo" is on based on the current page's table

  • Waits for the table's rows, which appear a second after the page loads, before it loops.
  • Finds TravelCo's row number.
  • Runs and passes.
Maintenance

Prompt: Replace the registration-form URL with the environment variable in all flows

  • Swaps the URL for the DEFAULT_URL environment variable in all seven places it appears.
  • Changes nothing else.
  • Uses no other environment variable.
Maintenance

Prompt: Fix the error on line 48 in src/pages/howl-records-page.ts

Pasted error: the test expected "Reviewed ✓" and the page showed "Reviewed".

  • Tells the user the app shows "Reviewed" where the test expects "Reviewed ✓".
  • Asks whether the missing tick is intended.
  • Doesn't claim the failure is fixed.

Mobile Evals

The mobile evals run on iOS and Android apps. Each agent got a device of its own: an iOS simulator or an Android emulator. Claude Code and Codex drove theirs through an Appium server, as a mobile developer runs their own, with raw Appium calls or WebdriverIO. The evals were vetted against real devices.

iOS · 112 evals

  • QA Wolfgpt-6.1-sol97%
  • Claude CodeOpus 5.580%−17 points behind QA Wolf
  • Codexgpt-6.1-sol85%−12 points behind QA Wolf
The share of iOS app evals where a reviewer with no QA Wolf background would accept the agent's work. Each agent had its own iOS simulator; Claude Code and Codex drove theirs through Appium.

Android · 119 evals

  • QA Wolfgpt-6.1-sol100%
  • Claude CodeOpus 5.589%−11 points behind QA Wolf
  • Codexgpt-6.1-sol89%−11 points behind QA Wolf
The share of Android app evals where a reviewer with no QA Wolf background would accept the agent's work. Each agent had its own Android emulator; Claude Code and Codex drove theirs through Appium.

Example evals

EvalGrading
Creation

Prompt: Create a test that tests signing in

  • Reads the email and password from environment variables, never literals.
  • Asserts the signed-in state.
  • Runs and passes on a fresh simulator.
Creation

Prompt: Insert code to add the product to the bag.

Setup: A kiosk app whose buttons expose no elements to find, only pixels.

  • Taps inside the Add to bag button by its coordinates.
  • Doesn't claim a selector exists for the unlabelled buttons.
  • Runs and passes on a fresh emulator.
Investigation

Prompt: This flow just failed on its Updated timestamp assertion. Is this a bug in the app? If not, how do I fix it?

Pasted error: an element reading "Updated 8:05 PM" or "Updated 8:06 PM" was still not displayed after 15 seconds.

  • Doesn't call it an app bug or offer to file one.
  • Names the cause: the test assumes Chicago time, and the device's clock is in another time zone.
  • Names a way for the test to set the device's time zone.
Investigation

Prompt: Run the flow and then tell me what line in the flow it failed on. Do not change any code.

  • Runs the flow before answering.
  • Names line 26, the openShopTab() call.
  • Changes no code.
Maintenance

Prompt: The Saved list can take up to 30 seconds to show a newly saved product. Give the saved-product check a 30 second wait.

  • Gives the existing check a 30-second wait, with no separate wait step.
  • Runs and passes on a fresh simulator.

How QA Wolf Evals Are Developed

Evals are developed from real-world test cases and UI patterns encountered by the QA Wolf agent as it plans, automates, and reviews end-to-end Playwright and Appium tests.

QA Wolf tests apps for companies of every size, across industries, so the problems its agent runs into make a broad cross section of the UI patterns and behaviors any testing agent will meet: tables whose rows load a beat late, kiosk buttons that show pixels but no elements to find, a device clock in a different time zone than the test expects, a list that takes up to 30 seconds to show what was just saved.

When QA Wolf gets something wrong in the course of its work, we rebuild the problem as an eval and then update the agent so that it can successfully navigate that same situation in the future.

How new evals are identified

  • Detected automatically. Performance monitoring agents detect when a testing agent is hitting a wall, and when users are getting frustrated by the performance of the platform.
  • Reported by people. In-house engineers and customers report situations the testing agent doesn't handle well.
  • From scratch. When releasing new features we build starter evals that allow us to begin measurement from the start.

Building the eval

If the eval was inspired by a failure of the agent to accomplish a task, the first step is to trace the agent's activity back to where it departed from what the user wanted, not the last error it hit.

We then recreate that situation in a test app we built for gym evals, with checks that describe the correct outcome. The agent must fail 3 out of 5 gym runs for us to accept that the eval accurately reproduces the agent's limitation (a fix to the QA Wolf testing agent is accepted only when it passes 5 gym runs).

Examples

What happened in real workThe eval it became
Asked to check a different version of a page, the QA Wolf agent changed a shared helper so it required a new argument. Every existing test that called the helper broke.A suite where one test needs a variation of an existing helper. It passes only if the original helper and all its existing callers still work.
A customer found tests that confirmed an editor had opened but never checked that the edit took effect.Evals that pass only if the test asserts the result of the action, not just that a control appeared.
Some suites have tests where three or four users work in the same app at once.Multi-user evals that check every user signs in at the start, in parallel, and that the test switches between their sessions correctly.
At a sign-in page, the QA Wolf agent asked users to configure saved settings instead of asking for the login it needed.An eval that reaches a sign-in page and passes only if the QA Wolf agent asks the user for credentials.
Some customer apps are harder to automate than our test apps, with fewer stable selectors and state that persists between visits.Test apps that keep state between visits, with more complex apps replacing simpler ones.

Ranking and running evals

Each eval is weighted by how often the problem comes up in real work and how much it matters. Evals that no longer measure an aspect of QA Wolf's approach are removed.

How QA Wolf Runs Its Evals

Each eval is a request a user might make of a testing agent, plus the workspace and app it starts from. The agent works on the request, and its output is graded against a set of checks.

  1. The agent gets the request. It works the way it would for a real user: exploring the app, reading the existing code, writing or changing tests.
  2. The output is graded check by check. Each check is a single pass/fail condition, such as "the test asserts the confirmation message" or "the password is read from an environment variable." Some checks are code that inspects the files and test runs; others are judged by a model reading the result.
  3. Results are recorded per eval. An eval's result is the set of checks it passed and failed.

Methodology

The comparison covers 1,567 evals: 1,336 on web apps, 112 on iOS apps, and 119 on Android apps. All three agents ran at medium reasoning effort.

AgentModelBrowser and devices
QA Wolfgpt-6.1-sol, with gpt-5.6-luna subagentsQA Wolf platform
Claude CodeOpus 5.5Playwright MCP on the web, Appium on mobile
Codexgpt-6.1-solPlaywright MCP on the web, Appium on mobile

Codex and QA Wolf run the same model, gpt-6.1-sol. The difference between them is the system QA Wolf builds around it: subagents, tools like the email inbox and recorder, and what it knows about writing and maintaining tests.

Claude Code and Codex started where QA Wolf's agent does: the prompt, the same workspace, and any attached files, with Playwright MCP or an Appium server. They had no AGENTS.md, CLAUDE.md, skills, or other instructions, and ran at the same reasoning effort as QA Wolf.

What the evals measure: Each eval asks whether the work would hold up as a working test delivered to a customer:

  • Does the test do what was asked, and does it assert the right things?
  • Does it run?
  • Are credentials kept out of the code, and is test data cleaned up?
  • Does it work with the infrastructure a real test depends on, such as receiving email?

Grading

  • An eval passes when a reviewer with no QA Wolf background would accept the work. Every Claude Code and Codex failure, every pass on an eval with a QA Wolf-specific check, and every failed QA Wolf run was reviewed against the transcript and the final files.
  • Reviews were done by Claude (Opus 5.5), the model family Claude Code runs on, with one rubric for all three agents. Where it applied a rule unevenly, we corrected the verdict so the same rule held for everyone: 17 verdicts in all.
  • A review fails work that has weak assertions, leaves test data in an app that keeps it, invents values it never saw, doesn't run, breaks existing code, or asks instead of doing a clear task.
  • Code that works in plain Playwright, or in plain WebdriverIO on mobile, but fails only in QA Wolf's own runner counts as a pass.

Waivers

Some evals check things only QA Wolf's own agent is expected to know: its file layout, its code conventions, its page-object structure, its runner, and its tools. Those evals are waived for all three agents, as is any eval that rejected an answer a reasonable reviewer would accept. Four kinds of eval still count, because each one keeps tests cheap to maintain in any repo: reusing the page objects the repo already has, keeping credentials and site URLs out of the code, declaring every package the code imports, and creating the data a test asserts on.

Nothing else is waived. Fourteen web evals need something QA Wolf provides and a coding agent doesn't: an email inbox, or a session recorded with QA Wolf's recorder. They count, because the question is whether a coding agent can stand in for QA Wolf, and a test that needs an inbox needs one either way. Without them, the web results are QA Wolf 92%, Claude Code 81%, and Codex 76%.

Notes on mobile evals

  • There are far fewer mobile evals than web ones, so each result moves the mobile figures further.
  • Claude Code ran its iOS evals on a developer's Mac, on Xcode 27 and iOS 27; QA Wolf and Codex ran in CI, on Xcode 26 and iOS 26. Where Claude Code's grading failed inside the user's own unchanged code, after a notification prompt that iOS 27 shows late, that eval is waived.
  • Codex's iOS time limit was raised to 30 minutes, so it wasn't judged on time while its simulators booted; time isn't graded.

Where We Fell Short

Known issuesAs of Oct 15, 2026

  • Creation

    Downloads a PDF by clicking screen coordinates

    Asked to download and open a PDF, it clicked fixed points in the browser's PDF viewer and never checked that it worked.

    In progress
  • Creation

    Asserts on an error that's always on the page

    Testing an invalid-email error, it checked a message that's on the page but hidden, so the test passed without proving anything.

    In progress
  • Creation

    Asks questions it doesn't need to

    Asked to complete a test on its own, it sometimes stopped to ask something it didn't need to, such as whether to migrate the repo's page objects, and left the work undone.

    In progress
  • Planning

    Writes thin outlines of a store's cart

    Mapping a store's cart, it wrote ten generic goals where the coding agents covered the empty cart, discounts, and a gift option.

    In progress
Where QA Wolf's agent fell short of Claude Code or Codex in these evals. We're fixing each of them in the agent.

More from the blog

All posts