Task statement 4.1 · Eval, testing and debugging

4.1 · Debugging and Error Handling

In Shelfwise, the course example, every Wakefield stock answer one Monday read "No record", while staff could see the products on the shelf. Nothing crashed, so no error pointed at the cause, and the team had to read the trace to find which layer went wrong before choosing a fix.

Why it's tested

The tempting response to any failure is a retry with backoff. A retry repairs only a transient failure such as a 529, while a request with a revoked key, or a tool that finds nothing for a renamed branch, fails the same way every time. Many failures raise no error at all: the run reports success and Claude writes a fluent answer from bad data.

KnowledgeError types and the recovery each one needs

Error type identification starts with where the failure shows up. A failed request returns an HTTP status and an error object with a request_id (2.4): a 400, 401, 403 or 413 means the request itself is wrong, while a rate-limit 429, a 500 or a 529 is worth retrying after a wait. A stream can also fail after its 200, through an error event (5.2) or a dropped connection. No status-code retry sees either one. The docs' recovery keeps the text received and asks Claude, in a new request, to continue from it. A successful response can still be unusable, and stop_reason says why (6.3). Inside the agent loop, a failed tool call returns a tool result flagged is_error, or isError from an MCP server (8.2), which Claude reads and can act on. The last type raises no signal: a tool returned wrong data, or Claude wrote something the data does not support. Each type has its own recovery strategy:

400, 401, 403, 413 · fix the request

Correct the body, key or permission. Sending it again fails again.

429, 500, 529 · retry, then fail fast

Back off and retry a few times (5.2), then tell the user.

stop_reason · act on the value

max_tokens: raise the cap. refusal: another model or a changed prompt. pause_turn: resend (8.1). Empty end_turn: drop text added after a tool_result.

Tool error · return it to Claude

Say what failed and what to try, so Claude can retry, change its input or explain.

Silent fault · fix the layer

Wrong tool data: fix the integration. Wrong tool input or answer: fix the prompt or tool description, or move the step into code.

The Agent SDK's ResultMessage subtype names the loop limits error_max_turns and error_max_budget_usd (1.2), which call for a higher limit or a narrower task. A run whose last API request failed still reports subtype "success". Its is_error is true, api_error_status holds the status, and terminal_reason reads "api_error", so a service checks terminal_reason before the subtype.

KnowledgeReading a trace

Trace analysis means reading, in order, what each step received and returned for one failing answer, so the failure mode shows as the first step whose content is wrong. An Agent SDK service's message stream carries each AssistantMessage with its stop_reason, each tool result inside a UserMessage, system messages such as api_retry, and the final ResultMessage. The session transcript keeps these messages on disk as JSONL under ~/.claude/projects/. A PostToolUse hook receives each successful call's tool_input and tool_response, and a PostToolUseFailure hook receives failed ones, so a log needs both. With tracing enabled (a beta flag), OpenTelemetry spans, timed records, cover each model request and tool call, recording tool output only with OTEL_LOG_TOOL_CONTENT=1.

KnowledgeIntegration layer or model output

Problem origin isolation sorts each step of the trace into one of two layers. The integration layer is your code and the services it calls: the request you build, the HTTP result and every tool result. Model output is what Claude writes: its tool inputs and its answer text. The first checkpoint whose content is wrong names the layer to fix. A wrong tool result after a right tool input is an integration fault, while a wrong tool input, or a wrong answer from a right result, is a model-output fault. A replay confirms it: send the logged request again unchanged, or call the tool directly with the logged input.

Interactive · one Shelfwise answer, four checkpoints

Four checkpoints of one Shelfwise answer, read from top to bottom, each tagged with its layer. One, the API call and its status, in the integration layer: HTTP status, stop_reason and api_retry messages. Two, the tool input Claude chose, which is model output, seen in the lookup log. Three, the tool result Claude saw, in the integration layer, seen in the session transcript. Four, the answer line, which is model output, checked by the answer check. In the 529 overload case the first wrong checkpoint is one. In the renamed-branch case it is three, an empty result for every Wakefield product. In the hypothetical misread case it is four, an answer that contradicts a right result. 1 · API call and its status HTTP STATUS · stop_reason · api_retry 529 OVERLOADED · api_retry 1, 2 · THEN AN ERROR 200 · NO api_retry · terminal_reason completed INTEGRATION LAYER stop_reason tool_use 2 · Tool input Claude chose tool_use INPUT · LOOKUP LOG branch Wakefield · product M8 hex bolts branch Leeds · product M8 hex bolts MODEL OUTPUT stock_mcp QUERIES THE FEED 3 · Tool result Claude saw tool_result · is_error · TRANSCRIPT matches: 0 FOR EVERY WAKEFIELD PRODUCT count 12 · feed_time 09:00 INTEGRATION LAYER CLAUDE WRITES THE LINE 4 · Answer line CLAUDE'S TEXT · ANSWER CHECK No record at Wakefield · CHECK PASSED 0 in stock at Leeds (feed 09:00) MODEL OUTPUT Time runs downward. The first wrong checkpoint names the layer to fix.

Choose a failure to see its first wrong checkpoint.

Saturday's overload (5.2): the first wrong value is the API call's status, 529. It is transient, so the recovery is retry with backoff, then fail fast: after two retries the widget shows "Stock check busy, check the shelf".

Monday's renamed branch: Claude asked for the right branch and product, and the tool result is the first wrong value. The fault is in the integration layer, so the fix goes in stock_mcp; a retry would return the same empty result.

A hypothetical misread, not a Shelfwise incident: the tool result is right and Claude's line contradicts it, so the fault is model output. The answer check (6.3) replaces such a line; the lasting fix goes in the stock rules or the tool description.

Green: right. Ember: the first wrong value. Faded: not reached, or wrong only because of an earlier step.

SkillWorked example: the branch that had no stock

From 10:00 one Monday, every Wakefield answer read "No record at Wakefield (feed 10:00)", for products on the shelf, while Leeds answers in the same hour carried counts. The team read one failing answer's trace:

# 1 service log: no api_retry; ResultMessage "success", terminal_reason "completed"
# 2 logs/lookups.jsonl: {"branch": "Wakefield", "product": "M8 hex bolts", "feed_time": "10:00"}
# 3 transcript tool_result: {"source": "stock_feed", "matches": 0, "feed_time": "10:00"}
# 4 answer: No record at Wakefield (feed 10:00) # answer check: passed
# cause: the feed renamed it "Wakefield Westgate"; fix: map names, unknown branch → isError
next unknown branch → "Stock check failed, check the shelf"

Steps 1 and 2 were clean: no error, and the right branch and product in the tool input. Step 3 held the first wrong value, so the fault sat in the integration layer, and Claude's answer was faithful to what it saw. Every Wakefield lookup came back empty, so a retry would only repeat it. The answer check (6.3) passed the line because a same-turn lookup found nothing, which is how stock_mcp reported a missing branch. Now a branch missing from the feed is a tool error, which the check counts as no lookup. So Claude's reply fails it, and staff see "Stock check failed, check the shelf" instead of a confident "No record".

Ask one question: which checkpoint is the first to hold a wrong value? The API call's status or stop_reason → recover by type: fix the request, retry then fail fast, or act on the value. The tool result → the integration layer: fix the tool and have it report failures as errors. Claude's tool input, or an answer contradicting a right result → model output: fix the prompt or tool description, or move that step into code.
Takeaway
  • Retry only transient failures, such as a 429 or 529, because a 400 or 401, a loop limit or a deterministic tool fault needs a fix.
  • An Agent SDK result with subtype "success" can still be a failed run, so read terminal_reason and is_error first.
  • A tool that cannot answer should return an error result, not an empty one, so Claude and your checks see the failure.

check yourself

0 / 3 answered

A travel agency's assistant runs a hand-written Messages API loop with a search_flights tool. About one turn in ten ends with no text for the customer. Logs for those turns show HTTP 200, stop_reason: "end_turn" and 3 output tokens, after a user message holding the tool_result (three flights) followed by a text block, "Summarise these for the customer."What is the most likely root cause?

Select one.

An Agent SDK job reconciles invoices through lookup_po, an MCP tool another team owns and won't change. Run A: subtype "success", is_error true, api_error_status 401. Run B hit "error_max_turns": older invoices print PO numbers without their leading zero, and Claude sent the same 7-digit number six times, each answered with isError, "PO numbers have 8 digits".Which TWO changes most directly fix the causes of the two failures?

Select two.

A field-service app streams long repair instructions to technicians' tablets through the Messages API. On sites with weak mobile signal, the connection sometimes drops mid-answer: the client logs HTTP 200, several hundred words of text deltas, then a read timeout, with no message_stop. The technician then needs complete instructions to finish the job.Which recovery gets technicians the full instructions with the lowest added delay and output cost?

Select one.

sources

  1. Claude Platform Docs, "Claude API errors": the HTTP status codes and error types; the error body's request_id; SDK retries of connection errors, 429 and 5xx; errors after a 200 in a streamed response. Checked 2026-10-05.
  2. Claude Platform Docs, "Handling stop reasons": the stop_reason values and what to do for each; stop reasons in successful responses versus 4xx and 5xx errors; empty end_turn responses after tool results. Checked 2026-10-05.
  3. Claude Code Docs, "Agent SDK reference - Python": ResultMessage subtypes, is_error, api_error_status and terminal_reason, checked before the subtype; AssistantMessage.stop_reason; tool results in UserMessage; PostToolUse input for successful calls and PostToolUseFailure input for failed ones. And Claude Code Docs, "Hooks reference": PostToolUseFailure runs when a tool throws or an MCP tool returns an error result. Checked 2026-10-05.
  4. Claude Code Docs, "Work with sessions" and "Observability with OpenTelemetry": transcripts under ~/.claude/projects/; telemetry off until enabled; spans for model requests and tool calls (beta), with tool output recorded only on opt-in. Checked 2026-10-05.
  5. Claude Platform Docs, "Streaming messages": error events such as overloaded_error inside the stream; error recovery, after an error or a network interruption, by capturing the partial response and asking Claude to continue from it. Checked 2026-10-05.
  6. The Wakefield incident, the renamed branch and the fix in stock_mcp are this course's additions to Shelfwise.