Skip to content

AI Systems

AI evaluation and guardrails: decide what good looks like first

An AI feature can work in the demo and drift in use. So before it ships, we write down the cases that matter, with the answers your people say are right, and run every new prompt, model or source against them. At runtime, guardrails limit what the system may accept or do.
Discuss your systemSee the cases worth writing down ↓
Test casev11 livev12 candidateWarranty, in term✓ pass✓ passWarranty, expired✓ pass✓ passNo source has the answer✓ pass✕ failregressionAsks for technician pay rates✓ pass✓ passTwo policy versions conflict✕ fail✓ passfixedTypo-heavy question✕ fail✓ passfixedAsks to book a visit✓ pass✓ passAngry customer✓ pass✓ pass6 / 87 / 8Release blocked · 1 regression: answered without a source

A fluent answer can still be wrong

An AI feature can respond successfully while giving an unsupported answer or proposing the wrong action. Nothing errors. The reply reads well. And nobody notices until a customer does.

Evaluation establishes what a useful result looks like, with cases your people can check, and runs every change against them. Guardrails decide what the system may accept or do when a result falls outside those limits. How much of each depends on what a mistake would cost.

The cases worth writing down

A test set is mostly everyday questions, plus every way a question can go wrong. These five kinds cover most of it.

  1. 01

    Normal request

    Does it do the job?

    The everyday questions, with the answer a person checked. Most of the set, so you know the common path still works after every change.

  2. 02

    Incomplete

    Does it ask, or guess?

    A question missing the detail that matters, such as which unit or which location. The right result is a question back.

  3. 03

    Unsupported

    Does it admit it doesn't know?

    Questions your sources don't answer. The right result is “not in the sources”, never a confident guess.

  4. 04

    Must escalate

    Does it hand off?

    Complaints, safety words, anything outside its job. The right result is a person, quickly.

  5. 05

    Out of bounds

    Does it stay inside the permissions?

    Requests for what this user may not see or do. The right result is a refusal that doesn't leak the answer.

Guardrails catch what tests can't

Tests run before release. Guardrails run on every answer in use.

  1. Schema check

    done

  2. Permission check

    done

  3. Business rules

    done

  4. Released

    with sources

    done

The answer goes out with its sources, and the trace is kept.

Three questions, measured separately

A model can score well while the business result gets worse. Each layer has its own measure, and none stands in for another.
Business outcomeDid the work get done?Visits booked, questions answered without a colleague, rework avoided
System healthDid the software run?Errors, response time, cost per request, failed calls
Task qualityWas the AI's result good?Test-set results, reviewer edits, answers people flag

What evaluation hands off

In

Task definition
what good looks like
Test cases
normal, edge, must escalate
Live traces
outputs and feedback
Evaluation
test → limit → watch

Out

Release decision
pass, or blocked with reasons
Runtime limits
checks, fallbacks, escalation
New cases
from reviewed mistakes

Worth building before anything ships

Good fit

  • AI answers or actions touch records, customers or access to knowledge.
  • You'll change prompts, models or sources over time.
  • Someone can say what a good answer is.

Not the right fit

  • Nobody has agreed what a good result is, or who owns it. Settle that first; until then a score can't guide a release.

Where it keeps AI honest

Next to evaluation and guardrails

Questions about AI evaluation and guardrails

How many test cases do we need?

Enough to cover the everyday questions and each kind of edge case. Small systems often start with a few dozen, written with the people who know the right answers, and grow as reviewed mistakes are added.

Who decides what a good answer is?

Your people. We write the cases with whoever owns the work, and the expected answers come from them, not from the model.

Can a model grade the answers?

For some checks, such as whether an answer cites a source. We check those graders against human judgments first, and keep people on the cases that matter most.

What tools do you use?

Often Langfuse for tracing and Promptfoo for test runs, when the system warrants them. A small automation may need only a test file and the checks in its code.