Step by Step
E
Eval types — exact match, contains check, LLM-as-judge, human
Exact match checks whether output equals a specific expected string; contains check verifies output includes a required phrase; LLM-as-judge uses a stronger LLM to rate outputs against a rubric (scalable); human evaluation is expensive but remains the gold standard.
Example: using LLM-as-judge to scalably rate hundreds of generated customer support responses against a quality rubric, reserving expensive human evaluation for a smaller, high-stakes sample.
V
Vibes are not evals — systematic measurement required
Deploying a prompt change based purely on it "feeling" better on a handful of manually-checked examples is not a substitute for systematic evaluation — a prompt that works on just 5 examples means essentially nothing about its true reliability.
Example: a developer feeling confident about a new prompt after manually checking just 5 examples, without realizing this provides essentially no statistical confidence about how the prompt will perform across the full range of real-world inputs.
A
Appropriate test set size — 50 to 500 examples
Before changing any production prompt, a test set of 50-500 representative inputs should be created to properly evaluate performance across a meaningfully broad range of real-world scenarios.
Example: creating a test set of 200 representative customer support queries, covering the range of question types the system is expected to actually handle in production.
L
A/B testing and regression testing — comparing and protecting
A/B testing compares the old and new prompt on the exact same evaluation set to determine which performs better; regression testing verifies that previously-passing "golden" examples still pass after any prompt change, protecting against accidentally breaking previously working behavior.
Example: A/B testing a new prompt against the current production prompt on the same 200-example test set, while also confirming a set of previously-passing golden examples still pass under the new prompt.
Applied Walkthrough
1
A developer creates a new version of a production prompt and manually checks it against 5 examples, all of which look good, feeling confident enough to deploy immediately.
2
Ask: is this sufficient evidence that the new prompt is actually reliable across the full range of real-world inputs? No — a prompt working on just 5 examples provides essentially no meaningful statistical confidence about its true performance.
3
Instead, the developer should create a proper test set of 50-500 representative examples, run the new prompt against this full set, and specifically A/B test it against the current production prompt on the same evaluation set.
4
They should also run regression testing to confirm previously-passing golden examples still pass under the new prompt, ensuring the change doesn't accidentally break previously working behavior before any production deployment.
Exam Application
Exams test whether you understand the specific eval types (exact match, contains check, LLM-as-judge, human evaluation) and whether you know the recommended test set size (50-500 representative examples) before deploying any prompt change, along with the practices of A/B testing and regression testing.
⚠ Common Trap
The most common trap is deploying a prompt change based on it feeling better across just a handful of manually-checked examples. This provides essentially no meaningful statistical confidence — proper evaluation requires a much larger, representative test set (50-500 examples) along with systematic A/B and regression testing before any production deployment.
✓ Quick Self-Check
1. Name one eval type mentioned in this lesson.
Exact match, contains check, LLM-as-judge, or human evaluation (any one).
Tap to reveal / hide
2. Why is checking a prompt on just 5 examples considered insufficient?
Because it provides essentially no meaningful statistical confidence about the prompt's true reliability across the full range of real-world inputs.
Tap to reveal / hide
3. What test set size is recommended before changing any production prompt?
50 to 500 representative examples.
Tap to reveal / hide
4. What does A/B testing compare in this context?
The old and new prompt versions, evaluated on the exact same evaluation set.
Tap to reveal / hide
5. What does regression testing verify?
That previously-passing "golden" examples still pass after a prompt change, protecting against accidentally breaking previously working behavior.
Tap to reveal / hide