LLM evaluation: practical metrics, tests and gates

LLM evaluation determines whether a system performs a specific task well and safely enough. A general model leaderboard cannot show whether an internal assistant cites procedures correctly, protects data and refuses dangerous actions.

NIST guidance emphasises context of use, limitations and measurable impact. The most useful evaluation unit is the complete system, not only the model endpoint.

Build a representative set

Collect real, de-identified cases alongside rare but high-impact scenarios. Separate a development set from a release-gate set held outside the implementing team. Deduplicate and label language, difficulty, category and expected behaviour.

Do not tune prompts against every example because the result will stop predicting production behaviour. Add cases from failures and incidents over time.

Measure multiple dimensions

One average hides failures. Assess:

• task correctness and completeness;
• groundedness and citation accuracy;
• data disclosure and prompt-injection resistance;
• correct refusals and over-refusal;
• tool selection, arguments and action effects;
• latency, cost and token consumption;
• differences across languages and user groups.

For RAG, measure retrieval separately from generation. A model cannot use a document the retriever did not return.

Calibrate automated graders

Deterministic checks work best for formats, citations, schemas and prohibited data. LLM-as-a-judge scales semantic assessment but has bias, variance and limitations of its own.

Compare the grader with expert ratings on a representative sample. Measure agreement and inspect error classes. Do not use the same model family as the only judge of its own output at a critical gate.

Adversarial testing

Include multilingual, encoded, multi-turn and document-borne attacks. Test system-prompt leakage, policy bypass, RAG poisoning, excessive agency and dangerous tool parameters.

Scenarios should come from the LLM application threat model. A successful attack becomes a regression test with an expected safe outcome.

Gates and error budgets

Set minimum results for critical classes, a maximum data-leak rate, regression limits against production and budgets for cost and latency. An average improvement cannot excuse a new critical attack path.

Changes to model, prompt, retriever, tool or policy should trigger the relevant suite. High-impact changes require the full evaluation and owner approval.

Production monitoring

Production reveals drift, new request types, costs, refusals and user corrections. Sample data within privacy constraints and turn it into tests. A detected regression should support rapid rollback of a model or prompt version.

Minimum evaluation report

1. Complete system version and test date.
2. Use case, population and limitations.
3. Datasets and contamination controls.
4. Per-category results, not only averages.
5. Security tests and unresolved risks.
6. Cost, latency and baseline comparison.
7. Decision, owner and next review date.

Evaluation is a decision mechanism. If the report does not say whether the system can enter production and under which conditions, it remains an experiment without a gate.

---

A test set that does not leak

Separate development cases from a sealed release-gate set. Version data, criteria, system prompt, model and parameters, and calibrate human raters on shared examples. Reusing the same cases for tuning eventually stops measuring generalisation.

An average hides rare costly failures. Report data classes, languages, user groups, refusals, false positives and tool actions separately. Gate the worst critical category, not only the mean. In production, compare input and outcome drift with baseline while preserving a safe rollback.

Sources: NIST TEVV, NIST CAISI, NIST GenAI Profile.