Lesson 4 of 4 in Context Management & Reliability

5.4 · Cost, latency, and reliability

Production choices between models, between streaming and batch, and around retries. This lesson covers model routing (cheap-fast triage, strong-reasoning execution), when to reach for the Batch API vs streaming, exponential backoff with jitter, and the reliability-first principle the exam consistently rewards.

Model routing

Don't use the biggest model for every step. Route cheap fast models to triage, simple turns, and pre-filtering; route the strong model to the actual reasoning. Mismatched model choice is a top cost mistake in production.

def handle(user_msg):
    # Step 1: cheap fast model triages intent
    triage = client.messages.create(
        model="claude-haiku-4-5",   # cheap, fast
        max_tokens=100,
        system=TRIAGE_PROMPT,       # classify intent
        messages=[{"role":"user","content":user_msg}]
    )
    intent = parse_intent(triage)

    # Step 2: simple intents get the cheap model
    if intent in {"greeting", "faq", "acknowledge"}:
        return client.messages.create(
            model="claude-haiku-4-5",
            ...
        )

    # Step 3: reasoning-heavy intents get the strong model
    return client.messages.create(
        model="claude-opus-4-8",    # strong, expensive
        ...
    )

# Typical result: 70-80% of turns handled by the cheap model,
# reserving the expensive model for what actually needs it

Batch vs streaming

Batch API for high-volume async jobs (significantly cheaper, higher latency). Streaming for low-latency user-facing UX. Match to the constraint that dominates: user waiting → stream; overnight batch job → batch.

Retry with backoff

Transient failures (rate limits, network hiccups) should retry with exponential backoff and jitter, up to a limit. Persistent failures should escalate, not retry indefinitely. Instrument to distinguish the two.

Good to know — 'Just add retry' when the root cause is a bug is not a fix; it's a delay.

Reliability first, cost second

Several exam scenarios present a cost-saving architecture that introduces a failure mode. The exam rewards reliability-first thinking. Cost optimization comes after the system works reliably, not before.

The three valid escalation triggers

This is a favorite exam distinction. Escalate to a human when: (1) the customer explicitly asks for one, (2) a policy gap exists — no rule covers the case, or (3) the agent is unable to make meaningful progress after reasonable attempts. NOT: detected customer frustration, model self-reported confidence below a threshold, task complexity, or number of iterations.

Good to know — Sentiment-based and confidence-based escalation are both anti-patterns that show up as wrong answers on the exam.
# Valid escalation logic
if customer_asked_for_human(request):
    escalate()
elif not policy.covers(request_type):
    escalate(reason="policy_gap")
elif no_progress_after_attempts >= 3:
    escalate(reason="stuck")

# INVALID escalation logic
if sentiment(request) == "angry":       # ❌ not a trigger
    escalate()
if model_confidence < 0.7:               # ❌ not a trigger
    escalate()

Takeaways

  • Route cheap models for triage, strong for reasoning
  • Batch for throughput, stream for UX
  • Retry transient failures with backoff; escalate persistent ones
  • Reliability first, cost second — always

Exam traps

Using the biggest model for every step
Route cheap fast models to triage; use the strong model for the actual reasoning turn. 70–80% of turns typically don't need the big model.
Routing blocking pre-merge checks through the Batch API
Batch is for latency-tolerant workloads. Anything a developer waits on must stay synchronous.
Escalating on sentiment or self-reported confidence
Valid escalation triggers are: explicit customer request, policy gap, or inability to make meaningful progress. Sentiment and self-reported confidence are not.

Practice scenario

An engineer proposes removing retry logic across the pipeline to save on retry costs, arguing 'most requests succeed on the first try anyway.' What should you flag?

← PreviousBack to domain