20% of exam score
Prompt Engineering & Structured Output
Where the exam gets tricky. Wrong answers sound right.
Prompt design, few-shot, chain-of-thought, extended thinking, XML structure, structured output via tool use, evaluation frameworks, prompt caching.
Key insight
Two words cover this entire domain: BE EXPLICIT. 'Be conservative' does not reduce false positives. Defining exactly which issues to flag — with concrete code examples for each severity level — does. Categorical criteria beat vague confidence thresholds every time.
The exam rewards
- Replace vague confidence instructions with specific categorical criteria: 'flag only when claimed behavior contradicts actual code behavior' outperforms 'be conservative'
- Few-shot examples (2-4 targeted cases showing reasoning for ambiguous calls) are more effective than detailed instructions for consistency
- tool_use with JSON schemas eliminates SYNTAX errors, but not SEMANTIC errors — schemas won't prevent wrong field values, mismatched sums, or fabricated data
- Make fields nullable when source documents may not contain the information — this prevents the model from fabricating required fields
- Batch API is for latency-tolerant workflows (overnight reports, nightly audits). Blocking pre-merge checks that developers wait on must stay synchronous
Common anti-patterns
- Using confidence thresholds ('only high-confidence findings') instead of categorical criteria
- Adding more instructions when 2-3 concrete examples would resolve the inconsistency
- Routing blocking pre-merge checks through the batch API
- Assuming JSON schemas prevent all extraction errors — they only prevent syntax errors