A new AI-powered QA tool seems to pop up every week. Making sense of all the different options can be confusing. How do they work? What do they do? Will they fit into my development process?

This guide categorizes AI testing tools into four types, helping you understand how each works and which fits your team's needs.

Before we begin, it's important to understand that unless the AI-powered QA tool is developing its own model from the ground up (a costly and extremely unlikely proposition), it's making use of someone else's underlying LLM from OpenAI, Anthropic, Google, etc.

The 4 types of AI testing tools

Rather than reviewing specific tools that change frequently, we've identified four main types you'll encounter:

1. Agentic Automated Testing: Generate deterministic test code from natural language prompts.
2. Agentic Manual Testing: Use adaptive locators and vision recognition to execute tests without exposing code.
3. IDE Co-pilots: Analyze your codebase to scaffold test code in your existing framework.
4. Session Recorders: Capture and replay browser sessions by instrumenting your application.

Each type applies LLM technology differently to the challenges of QA, leading to different outcomes in reliability, portability, and maintenance. We'll describe each approach at a high level, explain its benefits and drawbacks, and help you identify which fits your development process.

What is Agentic Automated Testing?

Agentic Automated Testing is an AI-powered testing tool that creates and maintains end-to-end tests as real code. It generates executable Playwright or Appium tests that run in your environment rather than inside a proprietary runtime. AI writes and updates the tests; code determines how they execute.

Example: QA Wolf

These AI agents take the best parts of AI and combine them with the best parts of traditional testing. Users prompt the AI to create a test ("Add pants to cart and check out"), which the AI generates in deterministic and verifiable Playwright or Appium code. As such, Agentic Automated Testing is considered the "gold standard" for AI-powered QA.

With Agentic Automated Testing, the efficiency and accessibility of natural language prompts return true E2E tests that are:

• Deterministic. Each test contains a series of steps, executed sequentially, and ending with an expected outcome.
• Verifiable. You can validate that the test was executed as intended each time.
• Realistic. The tests interact with the front-end UI the same way a human behind a keyboard would.

How self-healing works in Agentic Automated Testing

Self-healing is a fundamental feature of most AI testing tools since test maintenance (investigating failures and updating tests after a UI change) is the most time-consuming and labor-intensive part of the QA lifecycle. As we will see below, self-healing means different things depending on the type of AI agent we're talking about.

With Agentic Automated Testing QA Agents, self-healing means programmatically updating the Playwright or Appium test code. The AI identifies where a test is failing and makes the appropriate changes, such as new selectors or additional test steps.

Since the agent is repairing and validating deterministic code, there's no possibility for the agent to inadvertently allow a bug to escape the way there is with Agentic Manual Testing agents. The test code itself is what will determine whether the workflow passes or fails, not the judgment (or lack thereof) of an AI.

Benefits of Agentic Automated Testing

• True determinism. Agentic Automated Testing tools are the gold standard for AI-enabled E2E testing as they combine the speed and efficiency of AI with the determinism of coded tests.
• Test complexity. The benefit of using code instead of prompts is that code provides greater flexibility for more comprehensive testing (accessibility, performance, APIs, and complex tests like Canvas APIs and browser extensions). Of course, the exact testing capabilities will vary from vendor to vendor.
• Portability. Because the output is standard test code (e.g., Playwright, Appium), your tests aren't trapped inside a proprietary runtime. You can move them across environments, run them locally or in CI, and avoid vendor lock-in to a single vendor's infrastructure.
• Transparency. Every test is human-readable code, which makes it easy to review, audit for compliance, and trace exactly what the test does. Unlike opaque no-code flows, the logic is explicit.
• Extensibility. You can layer on custom helpers, utilities, and frameworks as your test suite matures. This allows teams to evolve their testing strategy without waiting for a vendor to add features.

Key takeaways: Agentic Automated Testing

• Generate verifiable Playwright or Appium tests that run in your environment. 
• Produce human-readable test code that can be reviewed, audited, and versioned.
• Update tests by modifying code directly rather than altering runtime behavior.
• Support complex testing beyond the browser, including APIs, mobile apps, and extensions.

What is Agentic Manual Testing?

Agentic Manual Testing tools use computer-use APIs from LLM providers. Each test step requires a call to an LLM, which analyzes the page or screen and determines how to proceed. The approach mimics human manual testers: the LLMs review written test plans and perform the specified actions. 

While the technology aims to make testing accessible to non-technical teams by removing code from the process, it comes with the same drawbacks as human manual testing: The approach is slow, the exact steps aren't documented for reproduction, it's expensive to use tokens, and the AI is limited to what it can see on the screen. Complex workflows, like API interactions or third-party integrations, are often out of reach. 

Additionally, LLMs are non-deterministic. The system decides how to execute each step in the moment, so the exact steps taken may change from run to run.

How self-healing works in Agentic Manual Testing

Self-healing in these systems means the tool adapts the test to keep it running—whether by swapping locators, falling back to vision, or trying alternate paths. That adaptability reduces maintenance, but it can also hide regressions, since the test may 'pass' even when the user experience is broken.

Benefits and drawbacks of Agentic Manual Testing

The main benefit of Agentic Manual Testing is its ability to side-step broken selectors by using adaptive locators, natural language mapping or computer vision to follow test steps. In theory, this makes tests easier to maintain—as the UI changes, the AI makes judgment calls.

And that's where you run into the drawbacks:

• Non-determinism. Agent-driven runs are less repeatable than code. Whether the system is choosing locators, exploring flows heuristically, or falling back to vision, you don't get the same guarantee of step-for-step consistency that Playwright or Cypress provides.
• Lack of portability. Some vendors offer an export-to-code feature, but the output is often verbose and loses key behaviors. Locator abstraction, vision-based matching, and other heuristics don't translate cleanly, so exported tests don't behave the same as they did inside the product.
• Vendor lock-in. The features that reduce maintenance—self-healing, exploration logic, adaptive locators—remain proprietary. Even with export options, the most valuable functionality stays tied to the vendor runtime, making it costly to switch.
• Coverage limitations. Agentic Manual Testing is built around browser interactions. They have limited reach into APIs, background jobs, third-party services, or state setup.
• High execution cost. Agentic Manual Testing is significantly more expensive to run than code-based automation, making large-scale or frequent execution cost-prohibitive.
• Slower performance. Tests executed through Agentic Manual Testing agents run much slower than compiled code, creating bottlenecks when integrated into fast-moving CI/CD pipelines.

Key takeaways: Agentic Manual Testing 

• Execute tests inside a proprietary runtime without exposing underlying test code.
• Rely on adaptive locators, heuristics, or vision to resolve UI changes at runtime.
• Limit coverage primarily to browser-based interactions.
• Reduce manual maintenance while increasing reliance on vendor-specific execution logic.
• Make exported tests difficult to reproduce outside the vendor environment.

What are IDE Co-pilots? 

IDE Co-pilots are AI-powered tools that generate test code directly from your application’s source code. They scaffold runnable tests in frameworks like Playwright, Cypress, or Jest that live in your repo, while leaving test execution, coverage decisions, and maintenance entirely with your team.

While AI-enhanced IDEs are not QA tools, strictly speaking, they are frequently considered by teams leveraging AI as part of their approach to automated testing. So we're including them here as an honorable mention.

Most of these IDE Co-pilots use the codebase itself as the source of truth to generate end-to-end tests. Instead of crawling the UI or recording flows, these tools analyze your application code and generate runnable tests in your chosen framework.

From the outside, it looks like a productivity boost: You prompt the tool or point it at a function, and it scaffolds test code that lives in your repo. Your team can run these tests in your existing CI/CD pipeline, and, because they are just regular test code, they are fully editable. 

Planning, coverage modeling, and assertions still fall to your team, but the tool can generate tests at any level of complexity your framework supports.

Risks and limitations of IDE Co-pilots

Despite their appeal, these tools come with important constraints:

• Implementation-driven coverage. Since they reason from your source code, they often verify what the code already does instead of whether the system meets business requirements.
• Conflict of interest. The codebase becomes both the specification and the subject of verification, which means the source of truth is also the source of the bugs.
• Ongoing maintenance. Like any test suite, you own execution, reporting, and upkeep. These tools won't self-heal—resilience depends on selectors and helpers your engineers create.
• Infrastructure requirements. While generating tests is straightforward, running them at scale (e.g., daily or on every pull request) requires CI/CD infrastructure that your team must build and maintain.
• Scaling challenges. Like any IDE, these tools do not cover the full testing workflow. They lack the infrastructure needed for parallel execution, environment management, and reliable debugging beyond a developer's local machine.
• Lack of visibility. These tools are designed for technical resources, leaving PMs, execs, and manual testers in the dark about test coverage, test results, and product quality.

Key takeaways: IDE Co-pilots

• Generate test scaffolding by analyzing the application source code.
• Produce editable test files that live in your repository and run in existing frameworks.
• Use the codebase as the primary source of truth for what gets tested.
• Leave execution, reporting, maintenance, and CI/CD integration to your team.
• Provide limited visibility into test coverage and results outside engineering.

What are Session Recorders?

Session Recorders are AI testing tools that capture and replay recorded browser sessions instead of executing deterministic tests. They record DOM events, user inputs, and network activity from real interactions and replay them inside a vendor-controlled environment rather than validating live end-to-end behavior.

Session Recorders don't "test" in the true sense of the word by interacting with the rendered UI and asserting that some result happened. Instead, Session Recorders have you instrument the code base (through a browser extension or code snippet on the application header) so they can execute lines of code directly.

To determine which lines of code to execute, the Session Recorders observe a human's clicks and keystrokes and log the network activity between the client and server. To run a test, a Session Recorder re-executes those recorded interactions against the application's UI, simulating real user behavior.

What's really happening under the hood is that they capture browser-rendered activity—DOM mutations, JavaScript events, user inputs, and network traffic—then reconstruct it on replay. To make this work, they typically mock or snapshot network calls (including to third-party services), which means they don't validate the actual backend or side effects.

Think of these tools like a driving simulator for websites. They're useful for replaying what a user saw, but not for true end-to-end testing. And because they rely on browser-level rendering, they won't work at all for native mobile apps or desktop applications (like Electron) that don't expose a DOM.

Benefits and drawbacks of Session Recorders

Setup is fairly straightforward: Just install an extension or code snippet on your application header, and you're off to the races. Tests can be developed by non-technical team members like designers, customer success managers, and product managers.

But there are drawbacks:

• New features remain untested until exercised. Test coverage only exists after someone has used the feature, which means bugs can slip through on first use.
• Not true E2E functional tests. It's nice that these things can stub out servers, but, of course, mocking can miss real-world issues like tricky redirects and cookies, cross-site rules, data mismatches, slow servers, and side effects (like an email that never actually gets sent). They're okay for reproducing buggy user sessions, but not so great for proving a given path works from beginning to end.
• Security concerns. Most testing tools require giving the vendor some level of access (for example, source code snippets, test results, or limited environment credentials). Session recorders go significantly further—they instrument the browser and continuously capture live sessions, which can include credentials, session tokens, internal URLs, and PII. Allowing a vendor that level of visibility into staging or behind-VPN environments greatly expands the attack surface, making access controls essential. There's also the ever-present possibility that the instrumentation in your code will get accidentally deployed to production.
• Limited test cases. These tools can only test what the application exposes in the UI. APIs, database calls, browser extensions, etc., would all be out of reach for these agents. Furthermore, they won't work for native mobile apps that don't render a DOM, desktop applications built in technologies that don't expose browser-level events (e.g., Electron, Qt, .NET), or heavily backend-driven workflows where correctness depends on server logic, state, or external APIs. Lastly, these tools won't let you test side effects (sending email, charging a credit card, generating a report, authentication, database state, emails, payments, and so on) that mocking can't prove.
• Potentially noisy results. These tools flag visual diffs in the application, which means that everything from intentional changes to feature flags to rendering errors in the environment could cause a test to fail. These tools can also create too many irrelevant low-impact tests, because they simply replay user flows, and there are nearly an infinite number of user flows.
• Vendor lock-in. The artifacts created by these tools are not open source and are only designed to be used within the tool.

Key takeaways: Session Recorders

• Capture and replay recorded browser sessions rather than validating live end-to-end behavior.
• Depend on application instrumentation and continuous session capture.
• Mock or snapshot network calls, which prevents validation of real backend side effects.
• Restrict testing to browser-rendered UI and DOM-level interactions.
• Create proprietary artifacts that are only usable within the vendor’s environment.

How to choose the right AI testing tool for your team

For any team using AI in QA, the decision comes down to what you want to optimize for—and what constraints you are willing to accept.

Choose Agentic Automated Testing QA Agents if your goal is:

• Deterministic, verifiable tests that catch real regressions
• Test portability across environments without vendor lock-in
• Coverage beyond the browser, including APIs, mobile apps, and complex workflows
• Human-readable test code suitable for audits and compliance

Choose Agentic Manual Testing QA Agents if your goal is:

• Fast setup with minimal coding
• Reduced test maintenance through adaptive locators or vision

And you are willing to accept:

• Non-deterministic test execution
• Browser-only coverage
• Proprietary runtimes and vendor lock-in
• Higher execution cost at scale

Choose IDE Co-pilots if your goal is:

• Faster test authoring for developers working inside the codebase
• Generating scaffolding in existing frameworks like Playwright or Cypress

And you are willing to accept:

• Implementation-driven coverage rather than business validation
• Full ownership of test execution, maintenance, and CI/CD infrastructure
• Limited visibility into test results outside the engineering team

Choose Session Recorders if your goal is:

• Replaying real user sessions to reproduce bugs
• Capturing visual diffs for UI debugging

And you are willing to accept:

• Instrumenting your application to capture live sessions
• Continuous access to session data, including credentials and tokens
• Browser-only validation with mocked or snapshotted backend behavior
• No verification of real side effects or backend correctness

Session recorders are fine if you just want replays, but they miss real-world issues, pile on noisy results, and box you into a vendor's sandbox. Agentic Manual Testing? It can be flashy, but the randomness and lock-in make them more of a gamble than a guarantee. IDE co-pilots? Great for scaffolding code, but they're really just parroting back your own source, and you're still left carrying all the maintenance.

Agentic Automated Testing agents are in a different league. They give you real, verifiable tests in code you own, with the speed of AI and the reliability of proven frameworks. No smoke, no mirrors, no lock-in—just tests you can see and trust.

For any team serious about leveraging AI in QA, the only logical choice is an Agentic Automated Testing tool—delivering the efficiency of AI without sacrificing the trustworthiness and durability that automated testing demands.