Task statement 4.1 · Eval, testing and debugging
4.1 · Debugging and Error Handling
In Shelfwise, the course example, every Wakefield stock answer one Monday read "No record", while staff could see the products on the shelf. Nothing crashed, so no error pointed at the cause, and the team had to read the trace to find which layer went wrong before choosing a fix.
share
The tempting response to any failure is a retry with backoff. A retry repairs only a transient failure such as a 529, while a request with a revoked key, or a tool that finds nothing for a renamed branch, fails the same way every time. Many failures raise no error at all: the run reports success and Claude writes a fluent answer from bad data.
KnowledgeError types and the recovery each one needs
Error type identification starts with where the failure shows up. A failed request returns an HTTP status and an error object with a request_id (2.4): a 400, 401, 403 or 413 means the request itself is wrong, while a rate-limit 429, a 500 or a 529 is worth retrying after a wait. A stream can also fail after its 200, through an error event (5.2) or a dropped connection. No status-code retry sees either one. The docs' recovery keeps the text received and asks Claude, in a new request, to continue from it. A successful response can still be unusable, and stop_reason says why (6.3). Inside the agent loop, a failed tool call returns a tool result flagged is_error, or isError from an MCP server (8.2), which Claude reads and can act on. The last type raises no signal: a tool returned wrong data, or Claude wrote something the data does not support. Each type has its own recovery strategy:
Correct the body, key or permission. Sending it again fails again.
Back off and retry a few times (5.2), then tell the user.
max_tokens: raise the cap. refusal: another model or a changed prompt. pause_turn: resend (8.1). Empty end_turn: drop text added after a tool_result.
Say what failed and what to try, so Claude can retry, change its input or explain.
Wrong tool data: fix the integration. Wrong tool input or answer: fix the prompt or tool description, or move the step into code.
The Agent SDK's ResultMessage subtype names the loop limits error_max_turns and error_max_budget_usd (1.2), which call for a higher limit or a narrower task. A run whose last API request failed still reports subtype "success". Its is_error is true, api_error_status holds the status, and terminal_reason reads "api_error", so a service checks terminal_reason before the subtype.
KnowledgeReading a trace
Trace analysis means reading, in order, what each step received and returned for one failing answer, so the failure mode shows as the first step whose content is wrong. An Agent SDK service's message stream carries each AssistantMessage with its stop_reason, each tool result inside a UserMessage, system messages such as api_retry, and the final ResultMessage. The session transcript keeps these messages on disk as JSONL under ~/.claude/projects/. A PostToolUse hook receives each successful call's tool_input and tool_response, and a PostToolUseFailure hook receives failed ones, so a log needs both. With tracing enabled (a beta flag), OpenTelemetry spans, timed records, cover each model request and tool call, recording tool output only with OTEL_LOG_TOOL_CONTENT=1.
KnowledgeIntegration layer or model output
Problem origin isolation sorts each step of the trace into one of two layers. The integration layer is your code and the services it calls: the request you build, the HTTP result and every tool result. Model output is what Claude writes: its tool inputs and its answer text. The first checkpoint whose content is wrong names the layer to fix. A wrong tool result after a right tool input is an integration fault, while a wrong tool input, or a wrong answer from a right result, is a model-output fault. A replay confirms it: send the logged request again unchanged, or call the tool directly with the logged input.
Interactive · one Shelfwise answer, four checkpoints
Choose a failure to see its first wrong checkpoint.
Saturday's overload (5.2): the first wrong value is the API call's status, 529. It is transient, so the recovery is retry with backoff, then fail fast: after two retries the widget shows "Stock check busy, check the shelf".
Monday's renamed branch: Claude asked for the right branch and product, and the tool result is the first wrong value. The fault is in the integration layer, so the fix goes in stock_mcp; a retry would return the same empty result.
A hypothetical misread, not a Shelfwise incident: the tool result is right and Claude's line contradicts it, so the fault is model output. The answer check (6.3) replaces such a line; the lasting fix goes in the stock rules or the tool description.
SkillWorked example: the branch that had no stock
From 10:00 one Monday, every Wakefield answer read "No record at Wakefield (feed 10:00)", for products on the shelf, while Leeds answers in the same hour carried counts. The team read one failing answer's trace:
# 1 service log: no api_retry; ResultMessage "success", terminal_reason "completed" # 2 logs/lookups.jsonl: {"branch": "Wakefield", "product": "M8 hex bolts", "feed_time": "10:00"} # 3 transcript tool_result: {"source": "stock_feed", "matches": 0, "feed_time": "10:00"} # 4 answer: No record at Wakefield (feed 10:00) # answer check: passed # cause: the feed renamed it "Wakefield Westgate"; fix: map names, unknown branch → isError next unknown branch → "Stock check failed, check the shelf"
Steps 1 and 2 were clean: no error, and the right branch and product in the tool input. Step 3 held the first wrong value, so the fault sat in the integration layer, and Claude's answer was faithful to what it saw. Every Wakefield lookup came back empty, so a retry would only repeat it. The answer check (6.3) passed the line because a same-turn lookup found nothing, which is how stock_mcp reported a missing branch. Now a branch missing from the feed is a tool error, which the check counts as no lookup. So Claude's reply fails it, and staff see "Stock check failed, check the shelf" instead of a confident "No record".
stop_reason → recover by type: fix the request, retry then fail fast, or act on the value. The tool result → the integration layer: fix the tool and have it report failures as errors. Claude's tool input, or an answer contradicting a right result → model output: fix the prompt or tool description, or move that step into code.- Retry only transient failures, such as a 429 or 529, because a 400 or 401, a loop limit or a deterministic tool fault needs a fix.
- An Agent SDK result with subtype
"success"can still be a failed run, so readterminal_reasonandis_errorfirst. - A tool that cannot answer should return an error result, not an empty one, so Claude and your checks see the failure.