Task statement 4.4 · Prompt engineering and structured output
Implement validation, retry, and feedback loops for extraction quality
Your extraction tool returns schema-valid invoice records, and your validator rejects about six in every hundred. Some rejections are misreads that one well-built retry fixes. Others need a value that was never in the input, and those fail however often you retry.
share
The tempting fixes all add attempts. "That was wrong, try again" makes the second attempt an independent guess, because the model is not told which check failed. A higher retry limit for a missing purchase order number only invites an invented value, because the number is in a document nobody sent.
KnowledgeValid is not the same as right
Tool use with a strict JSON schema eliminates schema syntax errors, such as malformed JSON or a missing required field. It cannot catch semantic validation errors, where every value has the right type but the record is still wrong: line items whose values don't sum to the total, or a VAT number placed in the wrong field. Only code that knows your business rules finds these, so your validator decides what reaches the ledger.
Eliminated by tool use
Owner: the strict schema. No retry needed.
Pass the schema, break a business rule
Owner: your validator, then a retry or a review.
More precise than the guide. The docs guarantee schema compliance "in most cases": output can break the schema when stop_reason is "max_tokens" or "refusal", and enum capitalisation is not guaranteed (checked 2026-10-01). Exam answer: tool use eliminates schema syntax errors, and semantic errors need your validator.
The loop, one rejected record at a time
The validator names each failure in a sentence the model can act on. Before retrying, your code asks whether the correct value is actually in what it sent, because a retry can only correct how the document was read.
- The extraction returns a schema-valid record whose values are unchecked.
- Your validator checks sums, formats and rules across fields.
- On a failure, your code asks whether the correct value is in what it sent.
- If it is, the retry sends the original document, the failed extraction and the specific errors, and the result is validated again.
- If it is absent or contradictory, the record keeps
nullor a conflict flag and goes to human review.
Interactive · triage a rejected record
Choose a rejected record, and see whether a retry can fix it.
Retry. The date is in the document; only its format is wrong. The retry names the ISO 8601 rule, and the corrected record passes.
Retry. calculated_total differs from stated_total and conflict_detected is false, so a line was misread. The error message gives the model both figures to reconcile.
Do not retry. The PO number is only in a master agreement that was never sent, so a retry can only return null again or invent a number. Keep null and route it, or add the agreement to the input.
Do not retry. The supplier's printed lines really do not add up, so conflict_detected is true. A retry could only choose one of two wrong numbers. Route it for review.
SkillA retry that gives the model something to correct against
Invoice INV-7731 prints a total of 1,410.00. The extraction records the installation line, printed as "2 × 270.00 = 540.00", as 270.00, so the lines sum to 1,140.00 and the validator rejects the record. A retry saying "That was wrong, try again" returned 1,140.00 twice, because nothing told the model which figure to check. The fixed request sends the invoice, the rejected record and the validator's message, so the model re-reads the lines, corrects installation to 540.00, and the record passes.
# Pydantic: a rule the schema cannot express @model_validator(mode="after") def lines_sum_to_total(self): s = sum(li.amount for li in self.line_items) if abs(s - self.stated_total) > 0.01: raise ValueError( f"line_items sum to {s:.2f}; " f"stated_total is {self.stated_total:.2f}") return self
<document> …the original invoice… </document> <previous_extraction> {"stated_total": 1410.00, "line_items": [ …, {"desc": "Installation", "amount": 270.00}, … ]} </previous_extraction> <validation_errors> - line_items sum to 1140.00; stated_total is 1410.00 </validation_errors> Re-read the line items against the document and correct the extraction.
The three parts can also travel as the next turn of the conversation: the document in the first message, the failed extraction as Claude's tool_use block, and the errors in a tool_result with is_error: true. The Messages API is stateless, so that request resends the document as part of the history.
KnowledgeWhen a retry cannot help
Retries succeed on format mismatches and structural output errors, because the right answer was in the input all along. A date extracted as "3rd March 2026" instead of ISO 8601 is fixed by a retry that names the rule. Retries are ineffective when the required information is absent from the source document. Invoice INV-8120 says "PO: per master agreement MA-2291", an external document the pipeline was never given, so each retry can only return null again or invent a number. Keep null and route the record, or add the agreement to the input.
SkillLet the extraction check itself
Extract the total twice: once as printed (stated_total) and once as the sum of the line items you extracted (calculated_total). Add a conflict_detected boolean that the model sets when the document's own figures disagree after it re-checks its reading. Your code compares the totals and acts on the flag.
// INV-7731: a line was misread → retry with the error {"stated_total": 1410.00, "calculated_total": 1140.00, "conflict_detected": false} // INV-9004: the printed lines add to 1,050.00 → review {"stated_total": 1150.00, "calculated_total": 1050.00, "conflict_detected": true}
A mismatch without the flag is a misread, so a retry is worth making. With the flag set, the invoice itself is inconsistent, and a retry could only choose one of two wrong numbers, so a person decides.
SkillTrack which pattern triggered each finding
A dismissal rate says the review bot is noisy, but not where. Add detected_pattern, a field naming the code construct that triggered the finding, to each structured finding, and store it with every dismissal. Grouping dismissals by that field shows which pattern produces false positives, so you know which criterion to rewrite.
{"file": "tests/db/teardown.ts", "line": 42,
"issue": "Empty catch block swallows errors",
"detected_pattern": "empty_catch_block"}
// dismissals by detected_pattern, last 14 days
empty_catch_block 41 of 48 dismissed
sql_string_concat 1 of 19 dismissed
Most dismissed catch blocks sit in test teardown code, where ignoring errors is deliberate, so that criterion is rewritten to skip teardown code and the SQL findings stay.
| Yes, but misread or misformatted | Retry with the document, the failed extraction and the specific errors. |
| No, it is absent or in an external document | Do not retry. Keep null and route it, or add the document. |
| The document contradicts itself | Do not retry. Set conflict_detected and route it. |
- A record that parses can still be wrong: semantic errors need your validator, not a stricter schema.
- A retry without the specific errors is a fresh guess, and one without the document has nothing to re-read.
detected_patternturns a dismissal rate into the name of the criterion to rewrite.