What is test automation?

Every release, someone has to make sure sign-up, login and checkout still work. Test automation, also called automation testing or automated testing, means writing those checks once as code, so a computer runs them on every change and reports anything that broke.

A person testing by hand has to start over after every deploy, and falls further behind as the product grows. An automated suite runs the same checks the same way every time, as often as you ship, and leaves people free for the testing that needs judgment.

What is test automation?

Test automation is using software to test software. Each automated test is a short program that puts the app in a known state, does something a user or another system would do, and checks the result. Those three parts are called Arrange-Act-Assert.

An automated checkout test, for example, might:

• Arrange: create a test account and put an item in its cart.
• Act: go to checkout and pay with a test card.
• Assert: check that the order confirmation appears and the order shows up in the account's history.

If any check fails, the test fails, and the run reports which step broke, with logs, screenshots or a video to show why.

"Test automation", "automation testing" and "automated testing" all mean the same thing. They're one part of software testing, alongside manual testing, where a person works through the app and judges what they see.

Automated tests usually run as part of a DevOps pipeline. When a developer opens a pull request or a build deploys to a test environment, the suite runs, and its result decides whether the change can ship.

Why is test automation important?

Software changes faster than people can retest it. Every release can break something that worked yesterday, and the more often a team ships, the more often that happens.

• Every change gets checked. Tests run on each pull request or deploy, so a regression shows up before customers see it.
• Developers hear about problems fast. A failing test points at the change that broke it while that change is still fresh in someone's mind.
• Releases stop waiting on QA cycles. A suite that runs its tests in parallel finishes in the time of its longest test, however many tests it has.
• The same check runs the same way every time. Code runs every step on every run, including on the days people are tired or rushed.
• Coverage can grow with the product. Adding a test costs far less than adding hours of manual testing to every release.

What should you automate?

Automate everything a user can do in your app. Start with the flows that would hurt most if they broke, get those tests stable, and keep adding tests until every flow has one. QA Wolf's guide to prioritizing your first 100 tests ranks flows by how many users touch them, whether they make money, what they depend on, whether they handle sensitive data and how recently their code changed.

Automate these first:

• Critical user flows: sign-up, login, checkout, payments and the core workflows customers pay for.
• Regression checks on features that already work and need to keep working.
• Flows that are slow or error-prone to test by hand, with many steps, several users, or many browsers and devices.
• Checks that need lots of data combinations, like pricing rules or permissions by role.
• Bugs you've fixed, so they can't come back unnoticed.

Keep these with people:

• Exploratory testing, where someone looks for problems nobody thought to write a test for.
• Usability and design, such as whether a screen is confusing or looks right.
• The first check of a new feature. Check it by hand while it still changes every day, then automate its test once the behavior settles.
• One-time checks, like a data migration you'll only run once.

People find new problems and decide what correct looks like, and automation makes sure those answers stay true on every release.

What are the types of automation testing?

Automated tests differ in how much of the system each one covers. Most teams use several kinds:

• Unit tests check one function or component on its own. They're fast, and developers write them alongside the code.
• Integration tests check that two or more parts work together, such as a service and its database, or an app and a third-party API.
• End-to-end tests drive the whole app the way a user does, through the real interface, from sign-up to checkout.
• Regression tests rerun existing checks after a change to confirm that nothing that worked before is broken.
• Smoke tests run right after a deploy to confirm the build starts and its critical pieces respond.
• Functional tests check that each feature does what its requirements say.
• Visual tests compare screenshots with an approved baseline to catch layout and styling changes.
• Cross-browser tests run the same checks in Chrome, Firefox and Safari's engine, WebKit.
• Accessibility tests check pages against standards such as WCAG.
• Performance, load and API tests measure speed under load, or check services directly without a user interface.

Several of these names describe the same tests from different angles. Many teams use "regression testing" and "end-to-end testing" interchangeably, because the end-to-end suite is the one they rerun after every change, and a smoke test is a small, fast slice of that suite that runs first.

A common way to balance the kinds is the test pyramid, with many fast unit tests at the bottom, fewer integration tests in the middle and a small set of end-to-end tests at the top. The pyramid keeps the top small because end-to-end tests used to be slow to run and costly to maintain. When the suite runs in parallel and AI agents maintain it, that cost drops, and end-to-end tests are the only ones that check what a customer actually does. So keep your unit tests, and give every user flow an end-to-end test as well.

What is a test automation framework?

A test automation framework is the library you write tests with. It drives the browser or app, finds elements on the screen, runs the tests and reports the results. Four frameworks cover most automated testing of user interfaces today:

• Playwright: Microsoft's open-source framework for web apps. It drives Chromium, Firefox and WebKit from one API, waits for elements automatically and runs tests in parallel, in JavaScript or TypeScript, Python, Java or .NET.
• Appium: the open-source framework for native and hybrid mobile apps on iOS and Android, built on the WebDriver protocol. Its Windows driver also automates Windows desktop apps. The best mobile app testing frameworks compares it with the alternatives.
• Selenium: the oldest of the four, built on the W3C WebDriver standard, with libraries for Java, Python, C#, Ruby and JavaScript. Many existing browser suites are written in it.
• Cypress: a JavaScript framework that runs tests inside the browser, popular with front-end developers for its interactive runner. QA Wolf doesn't recommend it for new suites, because Playwright does what Cypress does in more browsers, with parallel runs built in. QA Wolf explains why it chose Playwright over Cypress.

QA Wolf's AI agents write tests in the first two, as open-source Playwright and Appium code you own, so there's no lock-in.

Unit tests use their own frameworks, such as Jest or Vitest for JavaScript, pytest for Python and JUnit for Java. Around the framework sit the other tools a suite needs: a CI system to start runs, browsers or devices to run on, somewhere to see results and, more and more, AI testing tools that write and fix tests.

How do you start with test automation?

Start with the flows that would cost you most if they broke, and set up the habits that keep tests trustworthy before the suite gets big.

1. List your user flows. Write down everything users do in your app, and rank each flow by risk: how many users touch it, whether it makes money, what it depends on and how often its code changes.
2. Choose the level for each check. Business logic goes in unit tests. Anything that crosses services or the user interface gets an integration or end-to-end test.
3. Pick a framework that fits your stack. Use Playwright for web apps, Appium for mobile apps and your language's unit test framework for the code.
4. Write each test to stand on its own. Each test creates its own data and cleans up after itself, so tests can run in any order and all at once.
5. Run the tests on every change. Hook the suite into CI so it runs on each pull request or deploy, and let a failing test block the release.
6. Decide who investigates and maintains. Someone has to look at every failure and fix tests when the product changes, so plan for it before you have hundreds of tests.
7. Keep going until everything a user can do has a test. Once your critical flows are stable, add the next most important ones, and keep a list of which flows are covered so the gaps stay visible.

Where does test automation get hard?

Most of the work in test automation comes after the first tests are written. It goes into keeping hundreds of tests passing for the right reasons while the product changes every week.

Test maintenance

Tests encode how the app works today. When a button moves, a flow gains a step or the copy changes, the tests that touch it break, even though the product works. QA Wolf benchmarks show that half of all end-to-end tests need changes or removal within six weeks without ongoing maintenance. Tools with self-healing can fix some breaks on their own, but someone still has to check that each fix keeps what the test was meant to prove.

Flaky tests

A flaky test passes on one run and fails on the next with no change to the code. Common causes are timing, shared test data, unstable environments and tests competing for the same machine. Once a team has seen enough false alarms, it starts rerunning or ignoring failures, and real bugs slip through.

Infrastructure

Hundreds of end-to-end tests need browsers, devices and machines to run on. Most CI setups give you a fixed number of parallel workers, so a bigger suite means longer runs, and teams start running the full suite less often. Mobile and desktop apps add devices and virtual machines to manage.

Investigating failures

Every failure needs someone to work out whether it's a real bug, a broken test or a flake. Before AI agents took on the work, QA Wolf measured how long each step took its human QA engineers across 3.5 million tests. Creating a test took about 95 minutes, once. Then, every time they came up, investigating a failure took about 17 minutes, fixing a broken test 45 and reporting a bug 30. AI agents do each step much faster now, but investigation and maintenance still add up to more work than creation, because they keep coming back.

How does QA Wolf automate testing?

On QA Wolf's test automation platform, AI agents plan, build, run and maintain end-to-end tests for web, mobile and desktop apps, on QA Wolf's own testing infrastructure. When an agent needs an answer, such as whether a change was intended, it asks your team. From QA Wolf's docs and engineering posts:

• The agents map your app first. An agent explores your application and outlines the end-to-end flows to test, in Arrange-Act-Assert steps, so you can see what's covered, what isn't and where new features leave gaps.
• They write code you own. Describe a flow in plain language, and an agent writes it as a Playwright test for web apps or an Appium test for iOS, Android and Windows desktop apps. You can review, change and version the code like any other, and developers can ask for tests from Claude Code, Codex or another coding agent through QA Wolf MCP.
• Every test runs at once. Each test gets its own container, with its own browser, iOS or Android device or fresh Windows machine, so a suite finishes in the time of its longest test, whether it has 200 tests or 20,000. Runs start on every pull request, on a schedule or from your pipeline.
• Every failure gets investigated. A failing test is retried, then QA Wolf's AI starts investigating while the rest of the suite is still running, sorting each failure into a product bug, an out-of-date test or a flake. Real bugs reach you with a recording, logs and steps, in Slack or Teams and Jira or Linear.
• The agents maintain the tests. When your product changes, they update the tests it breaks, so the suite keeps passing.

On QA Wolf's full service, the same agents do the work and send their questions to QA Wolf's engineers instead of your team.

QA Wolf

0:10:00

Every test at once, each in its own container

Suite done

10 parallel nodes

2:33:00

10 tests at a time, 20 on each node

Suite done

200 tests that take 5 to 10 minutes each, run two ways. QA Wolf runs every test at once, each in its own container, so the suite is done in 10 minutes, the time of its longest test. On 10 parallel nodes, each node runs 20 of the tests one after another, so the suite takes about 2 and a half hours.

AutoTrader is a perfect example. Before QA Wolf, its manual testing was done by people pulled from the customer support team, and testing often took two or three days, sometimes a week. QA Wolf built, runs and maintains a suite of more than 500 automated tests, QA cycles dropped to 15 minutes, and AutoTrader avoided hiring six QA engineers, saving more than $600K a year.

Start now and ask the agent to map your app and build tests for your critical flows, or talk to sales.

The bottom line

Test automation turns the checks you'd repeat by hand into code that runs on every change. Start with the flows that matter most, get them stable and keep going until everything a user can do has a test. Write the tests in an open-source framework that fits your stack, and plan for the ongoing work of investigating failures and maintaining tests, because that work decides whether a suite stays useful.