Every team building on language models eventually arrives at the same uncomfortable question: is this actually good? The honest answer, for most teams, is that nobody knows. Quality is asserted in standups, defended with a handful of screenshots, and never measured. Then a prompt gets tweaked, something breaks quietly, and the first anyone hears about it is a complaint from a user three weeks later.

Evaluation is the difference between engineering and hoping. Here is a stack that works without requiring a research team to run it.

Start with the decision, not the metric

Before choosing any metric, write down the decision it will inform. There are usually only three:

  • Ship or do not ship. Is this good enough to put in front of users?
  • Better or worse. Did this change improve things, and by how much?
  • Where is it failing. Which slice of inputs is broken?

Metrics that inform none of these are decoration. A dashboard showing average response length is a decoration.

The golden set is the whole foundation

Collect one hundred to two hundred real inputs, from real users where possible, with expected outputs written by people who know the domain. Not synthetic questions generated by a model, which cluster around the same phrasing and quietly encode the same assumptions as your system.

Three properties matter more than size:

  • Representative. Sampled from actual usage, including the awkward inputs, not curated to the cases you already handle.
  • Stratified. Tagged by category, difficulty and expected behavior, so you can measure per slice. An aggregate score of 82% is much less useful than knowing you are at 94% on lookups and 51% on anything involving a date range.
  • Held out. A portion never used during development. If you tune against the whole set, you are measuring memorisation.

Include a deliberate set of inputs where the correct behavior is refusal: out of scope questions, questions the underlying data cannot answer, and adversarial prompts. Systems that score well on answerable questions and hallucinate on the rest are common and dangerous.

LLM-as-judge, with its biases named

Using a model to grade outputs is practical and scales, provided you treat the judge as an instrument that needs calibrating rather than an oracle.

Its known failure modes:

  • Length bias. Longer answers score higher, independent of correctness.
  • Position bias. In pairwise comparisons, the first option is favoured. Always run both orderings and average.
  • Self-preference. A model tends to prefer text generated in a style resembling its own.
  • Arithmetic blindness. Judges routinely mark numerically wrong answers as correct when the prose around them is confident.

Mitigations that work: score one specific dimension per call rather than asking for an overall rating, require the judge to quote the evidence for its score, use a coarse scale (a three or four point rubric beats a ten point one, where the difference between 6 and 7 is noise), and validate the judge against human labels on fifty examples before trusting it. Report that agreement figure alongside your scores, because it bounds how much any of them can be believed.

For anything with a verifiable answer, skip the judge entirely. Exact match, numeric tolerance, schema validation and unit tests are cheaper, faster and not subject to any of the above.

The metrics that correlate with trust

In production systems, three things predict whether users keep using a feature:

Faithfulness. Is every claim in the output supported by the provided context? This is the single strongest predictor of trust, because an unsupported claim that happens to be true is still a system that will eventually invent a false one.

Appropriate refusal. Does it decline when it should? Measure this explicitly as its own rate, on the refusal slice of your golden set. Systems optimized only for helpfulness drift toward answering everything.

Consistency. Does the same input produce a materially different answer on a second run? Run each golden case three times and measure variance. High variance means every quality number you have is a single sample from a wide distribution.

Notice that none of these are the benchmark scores model providers publish. Those tell you about general capability. They tell you nothing about your data.

Put it in CI or it will not happen

An evaluation suite that requires someone to remember to run it does not exist. The mechanics we use:

  • The suite runs on every change to a prompt, a retrieval parameter, a model version or a tool definition.
  • Results are written to a file committed alongside the change, so the diff shows the quality delta next to the code that caused it.
  • A threshold drop on any tracked metric fails the build. Overriding is allowed, with a written reason, because sometimes a deliberate trade is correct.
  • The full run is cheap enough to tolerate. A few hundred cases against a mid-tier model costs less than a coffee. If your suite is too expensive to run per commit, sample it per commit and run it in full nightly.

Version the prompts as ordinary code. Pin model versions explicitly and treat a provider's model update as a change requiring a full evaluation run, because it is one.

Production is the evaluation set you did not write

Offline evaluation catches regressions. It cannot tell you what users actually do, and users are inexhaustibly creative. Close the loop:

  • Log inputs, retrieved context, outputs and latency for every request, with sampling if volume demands it.
  • Give users a one-click way to flag a bad answer, and read the flags weekly.
  • Every flagged case that is genuinely wrong becomes a new golden set entry. The suite should grow monotonically, driven by real failures.
  • Track cost and latency per feature alongside quality, because a quality improvement that triples cost is a trade-off requiring a decision, not an unqualified win.

The honest summary

Evaluation will not tell you your system is good. It will tell you whether it is better than last week, where it is failing, and whether a change helped or hurt. That is enough to engineer with, and it is considerably more than most teams shipping LLM features currently have.

The teams that get this right are not the ones with the most sophisticated metrics. They are the ones who wrote down two hundred real examples early, ran them on every change, and were willing to believe the number when it disagreed with them.