Lesson 5 of 5 in Prompt Engineering & Structured Output

3.5 · Prompt caching for cost & latency

Prompt caching cuts cost and latency on long stable prefixes reused across calls. This lesson covers what benefits (long system prompts, big reference blocks, large few-shot sets), what breaks caching (dynamic content BEFORE the stable prefix), and when the write premium isn't worth it.

What caching does

Mark a stable prompt prefix as cacheable. On subsequent calls with the same prefix, the API reuses the cached compute at reduced cost and lower latency. The savings scale with prefix length.

What benefits most

Long stable system prompts. Large reference documents reused across queries. Big few-shot example sets. Anywhere you'd otherwise send the same tokens over and over.

# Mark the stable prefix cacheable with cache_control
r = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": LONG_STABLE_POLICY_TEXT,  # 20K tokens, reused
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[
        {"role": "user", "content": user_question}  # varies per call
    ]
)

# First call: writes to cache (small premium)
# Subsequent calls within TTL (~5 min): read from cache
# → ~90% cheaper on the cached portion, ~85% lower latency

Where caching breaks

Dynamic content that varies per call (user query, timestamp). Content inserted before a stable prefix. Frequent prompt tweaks during development. Caches expire (typically minutes), so low-traffic prompts may miss the cache anyway.

Good to know — If your 'stable' prompt has today's date at the top, nothing after it caches. Put dynamic content at the end.

Cost/latency tradeoff

Writing to the cache is more expensive per token than a normal read. You need enough cache hits to amortize the write cost. Rule of thumb: worth it for prompts hit at least 3x within the TTL window.

Takeaways

  • Cache long stable prefixes for cost + latency wins
  • Dynamic content must go at the end, not the top
  • Writes cost more; you need repeat hits to amortize
  • Cache TTL is short — low-traffic prompts may not benefit

Exam traps

Omitting the -p flag in CI
Without -p, Claude Code runs interactively and the CI job hangs waiting for input. -p is mandatory for pipelines.
Giving CI runs unrestricted bash access
Restrict allowed tools explicitly. CI running claude -p with open-ended shell is a security incident waiting to happen.
Piping Claude's output directly into a downstream shell
Treat Claude's output as untrusted input. Sanitize before passing to shells; escape before posting to comments.

Practice scenario

A GitHub Action runs 'claude' (without -p) to review a PR diff. The job times out after an hour. What's the fix?

← PreviousBack to domain