Most AI projects reach a moment where someone says: “It looks really good, let’s go live.” Usually that judgement rests on a dozen hand-picked examples tried in a meeting. Then the system meets real traffic, and the questions start. Why did it invent a delivery date? Why did it get worse after we changed the prompt? Is it better or worse than last month?
None of these questions can be answered without evaluations, or evals: a fixed set of test cases with known good answers, run automatically, every time something changes. Evals are to AI systems what automated tests are to software. You would not ship a payment service without tests; an AI system that reads invoices or answers customers deserves the same discipline.
Start from the work, not the model
A useful eval set comes from the process the system supports, not from generic benchmarks. Public leaderboards tell you how a model does on exam questions. They tell you nothing about whether it extracts the right due date from your suppliers’ invoices.
So we start by collecting real cases, anonymised where needed:
- Typical cases, in the proportion they actually occur. If 70% of incoming orders are simple reorders, 70% of the set should be too.
- Hard cases the experts know about: the supplier who puts the total on page two, the customer who writes in three languages.
- Cases where the right answer is “I don’t know”: missing data, out-of-scope questions, contradictory documents. A system that always answers is more dangerous than one that sometimes declines.
- Adversarial cases: inputs that try to make the system misbehave, such as an email that contains instructions to the AI.
Fifty well-chosen cases beat five hundred random ones. The set grows over time: every error found in production becomes a new test case, so the same mistake cannot quietly come back.
Measure what the business cares about
“Accuracy” sounds simple but hides choices. For an invoice-processing agent, a wrong amount is far worse than a missing purchase-order number. For a knowledge assistant, an answer without a source might be acceptable for internal FAQs but not for compliance questions.
We agree on metrics with the people who own the process, typically:
- Correctness per field or per answer, weighted by how much an error costs.
- Groundedness: is every statement backed by a source the system actually retrieved?
- Abstention quality: does the system decline when it should, and only then?
- Latency and cost per case, because a correct answer that takes a minute or costs a franc may still fail the business case.
Each metric gets a threshold that must be met before go-live. The threshold is written down before the results are known, so nobody is tempted to move the goalposts.
Automate the judging, then check the judge
Some answers can be checked mechanically: an extracted amount either matches or it does not. Free-text answers are harder. Here a second model can act as a grader, comparing the answer to a reference and a rubric. This works well, but only if the grader itself is validated: we have humans grade a sample, compare, and only trust the automated grader where it agrees with them.
Evals never stop
Going live is not the end of evaluation. Models are updated by their providers, documents change, users find new ways to ask. Three habits keep quality from drifting:
- Run the full eval set on every change: a new prompt, a new model version, a new retrieval setting. If a score drops, the change does not ship.
- Sample production traffic for regular human review, with personal data masked.
- Monitor signals that hint at trouble: rising abstention rates, more human corrections, longer answers, higher cost.
What this means in practice
Before a pilot goes live, you should be able to see a short report: the eval set, the agreed metrics, the thresholds and the results, with examples of where the system fails. If a supplier cannot show you that, the honest answer to “is it ready?” is “we don’t know yet”.
That report is also the best tool for the next decision. It shows which errors remain, what they would cost, and whether more work, a different model or a human-in-the-loop step is the cheapest way to fix them.