Lesson 4 of 5 in Prompt Engineering & Structured Output

3.4 · Evaluation — how you know it works

Prompt engineering without eval is guessing. This lesson covers building the held-out eval set, choosing grading strategies (exact-match, rubric, human), tracking regressions across changes, and the anti-patterns that make evals inflate scores.

Build the eval set first

Before you optimize the prompt, collect 20-100 real examples with known-good outputs. This is your ground truth. Every prompt change gets graded against this set. Without it, you're doing vibes-based development.

Grading strategies

Exact match for structured outputs (parse and compare). Field-by-field match for JSON. Rubric-based grading by a separate Claude call for free-form text. Human grading for genuinely ambiguous cases. Never use the same prompt to generate and grade — the grader shares the generator's assumptions.

Good to know — Same-model self-grading inflates scores. Use a different prompt at minimum, ideally a different model for critical evals.

Regression tracking

A prompt change that helps one case can break three others. Run the full eval set on every change, not just the case you were trying to fix. Improvements that don't show up on the full set usually weren't real improvements.

Change: added a new instruction to reduce hedging.
Before: 78% pass. After: 82% pass. Ship it — the number moved on the held-out set.

Categorical criteria beat vague confidence

A common exam trap: an instruction like 'be conservative' or 'only flag high-confidence issues.' This doesn't reduce false positives — it just shifts the model's tone. What works is CATEGORICAL criteria: precise definitions of what qualifies. 'Flag only when the claimed behavior contradicts observable code behavior' is testable and specific. 'Be careful' is not.

Good to know — Whenever the exam offers 'lower the confidence threshold' or 'ask the model to be more careful' as an answer, be suspicious. The correct fix is usually to make the criteria more specific.

❌ Vague:  'Only flag issues you are highly confident about.'
✅ Categorical: 'Flag an issue only if:
  - the diff introduces a call to a deprecated API, OR
  - the diff removes error handling from a public function, OR
  - the diff mutates a shared cache without a lock.
  Do NOT flag style, formatting, or naming.'

Anti-patterns in eval design

Building the eval set from the same examples the prompt was tuned on. Using metrics that don't correlate with what users care about. Grading only pass/fail when partial credit matters. Not versioning the eval set.

Takeaways

Build the eval set before optimizing
Separate generator from grader
Run the full set on every change
Metrics should reflect user-visible quality

Exam traps

Trying to enforce critical rules through CLAUDE.md

CLAUDE.md is prompt-level instruction — it's a suggestion. For rules that must not be violated, use hooks.

Assuming exit code 1 blocks the action

The block-with-feedback contract is exit code 2. Exit 0 allows; exit 2 blocks and feeds stderr back to Claude; other non-zero is an error but the action proceeds.

Using hooks for informational logging without exit 0

If your hook is only auditing, make sure it exits 0. Any non-zero exit signals a problem to Claude Code.

Practice scenario

A PostToolUse hook exits with code 2 and writes 'formatter failed' to stderr. What happens?

← Previous Next →