LLM Evaluation: How to Test LLM Apps

LLM evaluation checks what unit tests can't, such as whether an LLM feature tells a customer the right refund policy. You run the model, or the app built on it, against a set of test inputs, score each answer with code checks, a judge model or a person, and track the pass rate every time the prompt, model or code changes.

Teams call these test sets and their scorers "evals." They do the job of a regression suite for the parts of your product that don't give the same answer twice. They don't cover the product around the model: the screen the reply streams into, the record the assistant says it created, the button the user clicks next. That part needs end-to-end tests that drive the real app.

What is LLM evaluation?

LLM evaluation is measuring how well a large language model, or a system built on one, does the job you need it to do. It happens at two levels:

• Model evaluation compares models on general skills, using public benchmarks and leaderboards. It helps you choose a model.
• System evaluation tests your prompt, retrieval, tools and code together, on inputs from your own product. This is what most teams building on a hosted model mean by "evals."

Every eval has four parts: a dataset of test inputs, the system under test, a grader that scores each output, and a threshold that decides pass or fail. When any part of the system changes (the prompt, the model version, the retrieval settings), you rerun the dataset and compare the scores.

How is LLM evaluation different from software testing?

A normal test expects one right answer, but an LLM can answer the same input differently on every run, and several of those answers can be correct. So evals grade outputs against criteria and report a pass rate across many cases.

Failures can be partial, too, since an answer can be fluent, on topic and wrong about one fact. And the system can change without anyone touching your code, when the model behind your API call is updated.

You can make outputs more repeatable for testing by fixing a seed and lowering the temperature, which narrows the variation but doesn't remove the need to grade meaning.

What's the difference between offline and online evaluation?

Offline evaluation runs before release, on a fixed dataset, and online evaluation runs in production, on real traffic.

• Offline evals use test cases with known good answers or clear criteria, so they can check correctness. Use them to compare prompts and models, and as a regression gate before a change ships.
• Online evals score live responses, which have no reference answer. They rely on graders that don't need one (is the answer grounded in the retrieved documents, on topic, safe), on user feedback such as a thumbs-down, and on samples sent for human review.

The two feed each other, since a bad answer caught in production becomes a new offline test case that every later change is tested against. LangSmith's documentation, for example, describes both modes: offline evals on datasets for benchmarking and regression testing, and online evals on production traces for monitoring.

What metrics are used for LLM evaluation?

LLM evaluation metrics fall into four groups: code-based checks, similarity scores, model-graded scores and human ratings. Start with the cheapest one that can catch the failure you care about.

• Exact match and string checks. The output equals the expected label, contains a required phrase, leaves out a forbidden one, matches a regex or parses as JSON with the right fields. These checks are fast, free and deterministic. Use them for classification, extraction, structured output and anything the reply must or must never say.
• Semantic similarity. Turn the expected and actual answers into embeddings and check that their cosine similarity clears a threshold. It accepts different wording for the same meaning, but it measures closeness, so an answer can sit close to the reference and still get the one fact that matters wrong. Word-overlap scores such as BLEU and ROUGE, from translation and summarization research, have the same limit.
• LLM-as-a-judge. A second model grades the output against a rubric ("score the tone from 1 to 5") or picks from fixed labels ("correct, partly correct, wrong"). It handles open-ended answers that code can't check. Judges have biases of their own, so compare their grades with answers people have graded. QA Wolf's techniques for testing generative AI applications show how to limit a judge to multiple choice or a rubric and keep its scores stable.
• Groundedness and faithfulness. For retrieval-augmented generation (RAG), check that every claim in the answer is supported by the retrieved context. Pair it with retriever metrics such as context precision and context recall, so you can tell whether a bad answer came from bad search results or from the model. Ragas and DeepEval both include these metrics.
• Toxicity, bias and safety. Classifiers or judges flag toxic, biased or off-policy replies and leaked personal data. Red-teaming goes further, with prompts written to get around your guardrails.
• Cost and latency. Track tokens and response time for each case. A prompt change that doubles the tokens per answer is a regression too.

How do you build an LLM evaluation dataset?

Build it from real inputs. Pull questions from production logs and support tickets, then add the cases you're worried about:

• Common requests, in roughly the mix users send them.
• Edge cases: empty input, very long input, other languages, typos.
• Adversarial inputs: prompt injection, and requests the feature must refuse.
• Every past failure. Each bad answer someone reports becomes a test case.

Give each case a reference answer or the criteria a good answer meets, and tag it by category (billing, returns, account access) so a drop in one area doesn't hide inside a healthy average. Version the dataset with the prompt, so every score ties back to the cases it ran on. Cases written by a model can fill gaps, but read them before you trust them.

Hand-curated "golden" datasets go stale as prompts and user inputs change. QA Wolf's AI team samples production data at random instead of rebuilding a golden dataset for every change, and explains the trade-offs.

Weighting cases by how often they happen works for agents too. QA Wolf evaluates the AI agents that create and maintain its customers' tests on "gym scenarios" that simulate real test creation and maintenance work. Each scenario's weight comes from how often it occurs in real sessions, and running only the highest-weighted scenarios, about 5% of them, catches 95% of the problems.

When do LLM evals need human review?

Use people to set the standard and to check the automated graders, not to grade every run.

• Label a sample first. Have someone who knows the domain grade a few dozen outputs. Those labels define "good" and become the test for your LLM judge: if the judge often disagrees with them, fix the rubric before you trust its scores.
• Review production answers on a schedule, starting with low-scoring ones and ones users flagged. LangSmith, for example, has annotation queues for this.
• Keep a person in the loop for high-stakes answers, such as medical, legal or financial advice, where a wrong answer costs more than a slow one.

For how QA teams have approached this work on real AI projects, QA Wolf's reading list on testing generative AI applications collects experience reports worth reading.

How do you regression test prompts and model changes?

Treat the prompt, model version, temperature, retrieval settings and tool definitions as code, so any change to them reruns the eval set before it merges.

• Pin the model version where your provider offers dated versions, so the model can't change without a commit.
• Compare against a baseline, case by case. An average can stay flat while one category gets worse, so list the cases that flipped from pass to fail.
• Run each case more than once. One sample from a non-deterministic system proves little. Score the pass rate across runs, and treat a case that passes some runs and fails others as flaky until you know why.
• Rerun when the model changes, too. Switching models or moving to a newer version is a change like any other.

How do you run LLM evals in CI?

Run a fast, deterministic subset on every pull request, and the full set, model-graded checks included, nightly or before each release. Fail the build when the pass rate drops below your threshold or a must-pass case fails.

This minimal harness in Node needs no dependencies and no API key. The generate function stands in for your app's call to the model; replace it with the real call. Each case runs three times, because the output can vary, and the script exits with an error when the pass rate falls below 90%, which fails the CI job.

// eval.mjs: run a small eval set and fail the build if the pass rate drops.
// Run it with: node eval.mjs

// Stand-in for your app's call to the model. Replace it with the real call.
async function generate(input) {
  const replies = {
    "Classify this ticket: 'App crashes on login'": "bug",
    "Where is order 1042?": "Order 1042 shipped on Monday and should arrive Thursday.",
    "Return order 1042 as JSON": '{"orderId":"1042","status":"shipped"}',
  };
  return replies[input] ?? "";
}

const cases = [
  {
    name: "classifies a crash as a bug (exact match)",
    input: "Classify this ticket: 'App crashes on login'",
    check: (out) => out.trim().toLowerCase() === "bug",
  },
  {
    name: "names the order and promises no refund (contains / must not contain)",
    input: "Where is order 1042?",
    check: (out) => out.includes("1042") && !/refund/i.test(out),
  },
  {
    name: "returns valid JSON with the right fields (structure)",
    input: "Return order 1042 as JSON",
    check: (out) => {
      try {
        const order = JSON.parse(out);
        return order.orderId === "1042" && typeof order.status === "string";
      } catch {
        return false;
      }
    },
  },
];

const RUNS = 3; // model output varies, so run each case more than once
const THRESHOLD = 0.9; // the pass rate this change has to keep

let passed = 0;
for (const c of cases) {
  let casePasses = 0;
  for (let i = 0; i < RUNS; i++) {
    if (c.check(await generate(c.input))) casePasses++;
  }
  passed += casePasses;
  console.log(`${casePasses}/${RUNS}  ${c.name}`);
}

const rate = passed / (cases.length * RUNS);
console.log(`Pass rate ${Math.round(rate * 100)}% (threshold ${THRESHOLD * 100}%)`);
if (rate < THRESHOLD) process.exit(1);

As written, all three cases pass and the script exits with code 0. Change the order reply to mention a refund and the pass rate drops to 67%, so it exits with code 1 and the build fails.

Eval frameworks do the same job at scale. promptfoo reads test cases and assertions from a config file and can enforce minimum scores as a quality gate in CI. DeepEval plugs into pytest, so deepeval test run runs eval files the way pytest runs tests. Model-graded checks call an API on every run, so keep them out of the per-commit job if cost or speed matters.

Which tools are used for LLM evaluation?

Most teams use an eval framework to write and run test cases, and some add a platform that traces production traffic. Five commonly named options:

• promptfoo: an open-source CLI and library for evaluating and red-teaming LLM apps. Its assertions range from equals, contains, regex and is-json to embedding similarity, LLM rubrics and G-Eval.
• DeepEval: a Python framework that plugs into pytest. Most of its built-in metrics use an LLM as the judge, including G-Eval and faithfulness, and it has safety metrics such as toxicity, bias and PII leakage.
• Ragas: a library for evaluating LLM applications, with RAG metrics such as context precision, context recall, response relevancy and faithfulness.
• LangSmith: LangChain's platform for datasets, experiments and tracing, with offline and online evaluation and annotation queues for human review.
• OpenAI Evals: OpenAI's framework and registry of evals for testing models and systems built on them, published on GitHub. Evals can also be set up in OpenAI's dashboard.

How do you test an AI feature end to end in the real app?

Drive the product the way a user does, in a real browser, and check two things: what the app did and what the reply means. Evals call the model directly, so they never see the rest of the product.

Plenty can break in an AI feature without the model's wording changing at all:

• The reply streams in but never finishes rendering, or appears under the wrong message.
• The assistant says it created a ticket, booked a slot or applied a discount, and no record exists.
• The model gets the wrong user's context, an expired session or an empty document, and answers anyway.
• A guardrail or error message breaks the layout, or leaves the user with no way forward.

An end-to-end test catches these when it uses exact assertions for what must be exact: the record was saved, the order ID matches, the next button works. Use a semantic check, a narrow judgment by a model, only for the generated text, where many wordings are acceptable. Then run the tests on every deploy, like the rest of your suite.

How does QA Wolf test AI features?

QA Wolf tests AI features the way it tests the rest of an app: with end-to-end tests written in Playwright and Appium that run in a real browser or on a device, and AI agents that investigate every failure. Two kinds of assertion cover the generated text.

• Semantic assertions. ai.expect(locator).toSatisfy(statement) passes when an element's text means what the statement says, so "Your order has shipped" and "The parcel is on its way" can both pass a check for a shipping confirmation. QA Wolf evaluates each one with Jev, TypeSafe's model, so the test needs no API key. The default pass threshold is a 0.9 probability, and an uncertain judgment fails the assertion. Semantic assertions run in web flows.
• Judge-model assertions. For a fuller grade, a helper sends the prompt and the reply to Claude or ChatGPT, using your API key, and gets back a structured verdict: whether the reply satisfies the prompt, a score from 0 to 1, and lists of missing requirements, incorrect information and safety concerns. The same test can assert on token counts, so a prompt change that inflates cost fails too.

In QA Wolf's example of a semantic assertion in a support flow, Playwright checks the facts first: the test confirms the API saved an open ticket for the right order in the delivery queue, then asks Jev only whether the reply tells the customer the issue went to the delivery team. If no ticket exists, the test fails no matter how convincing the reply sounds. When a test fails, QA Wolf's agents investigate it, and only real bugs reach your team. If an agent needs a question answered, it asks you.

Hirevue, an AI-powered hiring platform, is a perfect example. It doubled its release velocity with QA Wolf, moving from a release every two weeks to one every week. Start now and ask the agent to write end-to-end tests for your AI features, or talk to sales.