Think back to high school science: in every controlled experiment, you change one variable and keep everything else the same. In end-to-end (E2E) testing, the term for that is determinism: you change the application build while keeping the test method the same. If neither the application nor the test changes, you should get the same result each time the test runs. If the test is failing (after you’ve confirmed it wasn’t a flake), then you know there’s a bug in the application.
Now, determinism isn’t the whole story. Test results also have to be interpretable. A QA engineer or developer needs to see what steps the test took, the data used, the assertions, and how the system performed. Otherwise, all you have is a pass/fail result and a mountain of work to manually reproduce each test.
This is why we built QA Wolf’s AI to operate the QA Wolf platform, a code-generating system that converts application behavior into executable, version-controlled tests instead of a codeless AI that follows English-language prompts.
How codeless AI breaks the QA experiment
Codeless AI tools promise to make testing faster and easier to maintain, but going faster doesn’t do your team any good if the tests aren't understandable or even reliable. The point of testing is to give teams the information they need to fix the bugs and the confidence to release new features. Each time a codeless agentic AI system runs, the AI decides what to test, how to run it, and what counts as a pass or fail, with the possibility of making different choices from one run to the next. Further, and even more dangerously, because the test runs are inconsistent, the artifacts they generate (screenshots, dashboards, etc.) can only tell you what happened. Not why it happened or how the AI decided a test should pass or fail.
Let’s break down the problem with codeless AI:
Hallucinated test steps. In science, if you change your testing procedure between experiments, you corrupt the data and the results. Codeless AI tools often change their own test steps—skipping validations, adding irrelevant steps, or “auto-healing” selectors when the UI changes. Each change redefines what the test measures. The application and the test both move at once, so any result could be caused by either. What appears to be a valid outcome is no longer just an isolated observation of app behavior; it’s a moving target.
Black boxes with no ownership or accountability. An experiment without notes can’t be verified. Many codeless systems provide only surface-level outcomes—a pass, a fail, or a screenshot—without preserving the steps, data, or assertions that produced them. They also remove clear authorship: no QA engineer can confirm that the AI tested what it was supposed to test.
Running to pass instead of running to validate. Some codeless AI testing systems operate from the principle that if a test can pass, then it should pass. Which means that they will adjust the test to achieve a passing result, rather than validating the intended behavior.
These aren’t isolated quirks; they’re “features” of codeless AI systems—design choices that trade clarity and control for the illusion of simplicity. By hiding the complexity behind automation, they leave teams with no way to understand, reproduce, or defend their results.
Rebuilding the QA Experiment
QA only works when the integrity of the tests is preserved on each run, which is why our AI tests with Playwright and Appium.
Code keeps the process deterministic. Each test step can be defined explicitly in code, so the same actions and checks run the same way each time. When the build, environment, and data stay consistent, identical runs yield the same results. The application can change, but the test logic doesn’t.
Artifacts make results interpretable. Code-based tests preserve that interpretability by producing readable logic, version history, logs, and traces. When something fails, teams can trace what ran and why, making the results meaningful as evidence.
When determinism and interpretability exist together, testing works as a controlled experiment again. Results no longer depend on hidden logic or evolving behavior. They can be seen, understood, and trusted as evidence of product behavior.
At QA Wolf, that’s exactly the system we’ve built. Every test is real, executable code. Every run produces visible artifacts. Every result can be traced, read, and understood. The product can evolve, but the logic that tests it doesn’t. That’s how we keep testing an experiment instead of a performance—and how teams using QA Wolf can move fast without guessing what their results mean.
.svg){kind=logo}
