Unit tests assume deterministic output. LLM output varies by design, even at temperature 0, because every model version shifts the distribution. LLM evals in CI/CD close that gap with four pieces: a golden dataset, deterministic checks, an automated judge, and a regression gate in the pipeline.

The golden dataset comes first

Hand-curate 50 to 200 input/expected-output pairs, harvested from real user queries and production failures. Every incident contributes two or three fresh cases. Without a golden set, every quality debate turns into opinion combat.

Version it as a fixture next to the code (evals/golden.yaml). A dataset living outside the repo dies forgotten within a sprint.

Deterministic checks before judges

Run the zero-token layer first:

  • JSON validity and schema conformance via zod or pydantic
  • Forbidden phrase lists (competitor names, legal promises)
  • Length ceilings and mandatory output shapes

This layer catches about 60% of regressions for free. Judges handle the remainder.

LLM-as-judge with a written rubric

A strong model scores each output against a documented rubric: factual correctness, tone, format adherence, a 1-5 grade per criterion. It costs 10x less than human review. Every judge carries bias; audit 5% of grades by hand to keep the scoring honest.

Regression gates in CI

The suite runs on every prompt or model bump (GitHub Actions, GitLab CI, whichever you run). A 2-3 point drop in composite score blocks the merge; in practice that threshold flags real regressions. With promptfoo, promptfoo eval --config promptfooconfig.yaml becomes a single workflow step, with output diffs on the pull request.

Production drift monitoring

PR-time evals protect the present; drift shows up later. Providers swap weights without notice. Sample 1-2% of live traffic every night, score against baselines, and page the on-call after three consecutive days of decline. Your eval suite is the tripwire.

Tooling by team size

promptfoo covers open-source pipelines with YAML configs and output comparison on pull requests. Braintrust and LangSmith ship managed platforms with tracing and online datasets. Small teams go far with plain pytest fixtures and review discipline.

Start with 30 golden cases, three deterministic checks, and a two-point merge gate. One afternoon of setup buys permanent protection against the next breaking model swap.