Lesson 6 of 6 in Agentic Architecture & Orchestration

1.6 · Failure modes & debugging agents

Agents fail in ways that traditional software doesn't. They loop on the same tool, silently claim success, drift as context grows, and produce different behaviour on identical inputs. This lesson covers the five recurring failure modes, why per-iteration logging is non-negotiable, and why chasing determinism is the wrong instinct.

The five ways agents fail

1) Tool selection: the agent picks the wrong tool. 2) Tool loop: same tool called repeatedly with same args. 3) Context degradation: long context, poor recent decisions. 4) Silent success: the agent claims to have done something it didn't. 5) Runaway iteration: the loop won't terminate.

Logs are the primary debugger

Log every iteration: input messages length, tool calls made, tool results, stop_reason, cumulative cost, elapsed time. Without this you're guessing. Agents are inherently stochastic; you need the trace to see patterns.

Reproducibility is expensive

Setting temperature to 0 doesn't make agents deterministic. Tool results, timing, and message ordering all vary. Instead of chasing reproducibility, run enough trials to see the distribution of behaviors, then fix the failure modes that appear.

Run the same request 20 times, note how often each failure mode shows up, prioritize the top one.

The eval-first workflow

Build an eval set of representative inputs with known-good behaviors before you tune. Grade each change against the eval set. Improvements that don't show up on the eval set usually aren't real improvements.

Good to know — Vibes-based iteration on agents produces confident engineers and unreliable systems.

Takeaways

  • The five failure modes: wrong tool, tool loop, context drift, silent success, runaway
  • Rich per-iteration logs are non-negotiable
  • Don't chase determinism; measure distributions
  • Evals before optimization

Exam traps

Chasing full determinism with temperature=0
Temperature=0 does not make agents deterministic. Tool result ordering, timing, and API-side variability all vary. Measure distributions of behaviour instead.
Debugging without per-iteration logs
Without a rich trace of every iteration (input messages, tool calls, tool results, stop_reason, cost), you're guessing. Agents are stochastic; you need the trace to see patterns.
Fixing prompts based on vibes instead of an eval set
A prompt change that helped on the one case you looked at can break three you didn't. Build the eval set first; grade every change against it.

Practice scenario

An agent occasionally 'succeeds' by claiming a refund was issued when the issue_refund tool actually failed silently. Which of the following is the correct diagnostic and fix?

← PreviousBack to domain