Task statement 4.4 · Prompt engineering and structured output

Implement validation, retry, and feedback loops for extraction quality

Your extraction tool returns schema-valid invoice records, and your validator rejects about six in every hundred. Some rejections are misreads that one well-built retry fixes. Others need a value that was never in the input, and those fail however often you retry.

Why it's tested

The tempting fixes all add attempts. "That was wrong, try again" makes the second attempt an independent guess, because the model is not told which check failed. A higher retry limit for a missing purchase order number only invites an invented value, because the number is in a document nobody sent.

KnowledgeValid is not the same as right

Tool use with a strict JSON schema eliminates schema syntax errors, such as malformed JSON or a missing required field. It cannot catch semantic validation errors, where every value has the right type but the record is still wrong: line items whose values don't sum to the total, or a VAT number placed in the wrong field. Only code that knows your business rules finds these, so your validator decides what reaches the ledger.

Schema syntax errors

Eliminated by tool use

Owner: the strict schema. No retry needed.

Semantic validation errors

Pass the schema, break a business rule

Owner: your validator, then a retry or a review.

More precise than the guide. The docs guarantee schema compliance "in most cases": output can break the schema when stop_reason is "max_tokens" or "refusal", and enum capitalisation is not guaranteed (checked 2026-10-01). Exam answer: tool use eliminates schema syntax errors, and semantic errors need your validator.

The loop, one rejected record at a time

The validator names each failure in a sentence the model can act on. Before retrying, your code asks whether the correct value is actually in what it sent, because a retry can only correct how the document was read.

  1. The extraction returns a schema-valid record whose values are unchecked.
  2. Your validator checks sums, formats and rules across fields.
  3. On a failure, your code asks whether the correct value is in what it sent.
  4. If it is, the retry sends the original document, the failed extraction and the specific errors, and the result is validated again.
  5. If it is absent or contradictory, the record keeps null or a conflict flag and goes to human review.

Interactive · triage a rejected record

Flow of a validation-retry loop: a schema-valid record goes to your validator; passing records go to the ledger; failing records are retried with specific errors only when the correct value is present in the input, otherwise they are flagged and routed to human review extracted record SCHEMA-VALID · VALUES UNCHECKED block.input validator · your code SUMS · FORMATS · CROSS-FIELD RULES passes ledger RECORD ACCEPTED fails · specific errors Is the correct value in what was sent? yes · format, structure, a misread retry request original document + failed extraction + the specific validation errors no · absent or contradictory do not retry keep null, or set conflict_detected: true, and route the record to human review retry → validate again A retry fixes how the document was read. It cannot fix what the document says, or add a value that was never sent.

Choose a rejected record, and see whether a retry can fix it.

Retry. The date is in the document; only its format is wrong. The retry names the ISO 8601 rule, and the corrected record passes.

Retry. calculated_total differs from stated_total and conflict_detected is false, so a line was misread. The error message gives the model both figures to reconcile.

Do not retry. The PO number is only in a master agreement that was never sent, so a retry can only return null again or invent a number. Keep null and route it, or add the agreement to the input.

Do not retry. The supplier's printed lines really do not add up, so conflict_detected is true. A retry could only choose one of two wrong numbers. Route it for review.

Time reads downward. The dashed rail is the retry loop, which carries a corrected record back to the same validator. Only the left branch takes it.

SkillA retry that gives the model something to correct against

Invoice INV-7731 prints a total of 1,410.00. The extraction records the installation line, printed as "2 × 270.00 = 540.00", as 270.00, so the lines sum to 1,140.00 and the validator rejects the record. A retry saying "That was wrong, try again" returned 1,140.00 twice, because nothing told the model which figure to check. The fixed request sends the invoice, the rejected record and the validator's message, so the model re-reads the lines, corrects installation to 540.00, and the record passes.

The validator names the failure
# Pydantic: a rule the schema cannot express
@model_validator(mode="after")
def lines_sum_to_total(self):
    s = sum(li.amount for li in self.line_items)
    if abs(s - self.stated_total) > 0.01:
        raise ValueError(
          f"line_items sum to {s:.2f}; "
          f"stated_total is {self.stated_total:.2f}")
    return self
The follow-up request carries three things
<document> …the original invoice… </document>

<previous_extraction>
{"stated_total": 1410.00,
 "line_items": [ …,
   {"desc": "Installation",
    "amount": 270.00}, … ]}
</previous_extraction>

<validation_errors>
- line_items sum to 1140.00;
  stated_total is 1410.00
</validation_errors>

Re-read the line items against the
document and correct the extraction.

The three parts can also travel as the next turn of the conversation: the document in the first message, the failed extraction as Claude's tool_use block, and the errors in a tool_result with is_error: true. The Messages API is stateless, so that request resends the document as part of the history.

KnowledgeWhen a retry cannot help

Retries succeed on format mismatches and structural output errors, because the right answer was in the input all along. A date extracted as "3rd March 2026" instead of ISO 8601 is fixed by a retry that names the rule. Retries are ineffective when the required information is absent from the source document. Invoice INV-8120 says "PO: per master agreement MA-2291", an external document the pipeline was never given, so each retry can only return null again or invent a number. Keep null and route the record, or add the agreement to the input.

SkillLet the extraction check itself

Extract the total twice: once as printed (stated_total) and once as the sum of the line items you extracted (calculated_total). Add a conflict_detected boolean that the model sets when the document's own figures disagree after it re-checks its reading. Your code compares the totals and acts on the flag.

// INV-7731: a line was misread → retry with the error
{"stated_total": 1410.00, "calculated_total": 1140.00,
 "conflict_detected": false}
// INV-9004: the printed lines add to 1,050.00 → review
{"stated_total": 1150.00, "calculated_total": 1050.00,
 "conflict_detected": true}

A mismatch without the flag is a misread, so a retry is worth making. With the flag set, the invoice itself is inconsistent, and a retry could only choose one of two wrong numbers, so a person decides.

SkillTrack which pattern triggered each finding

Scenario switchThis skill runs on the Claude Code for Continuous Integration scenario, because the guide writes the feedback loop in code-review terms.

A dismissal rate says the review bot is noisy, but not where. Add detected_pattern, a field naming the code construct that triggered the finding, to each structured finding, and store it with every dismissal. Grouping dismissals by that field shows which pattern produces false positives, so you know which criterion to rewrite.

{"file": "tests/db/teardown.ts", "line": 42,
 "issue": "Empty catch block swallows errors",
 "detected_pattern": "empty_catch_block"}
// dismissals by detected_pattern, last 14 days
empty_catch_block    41 of 48 dismissed
sql_string_concat     1 of 19 dismissed

Most dismissed catch blocks sit in test teardown code, where ignoring errors is deliberate, so that criterion is rewritten to skip teardown code and the SQL findings stay.

Before any retry, ask: Is the correct value in what you sent the model?
Yes, but misread or misformattedRetry with the document, the failed extraction and the specific errors.
No, it is absent or in an external documentDo not retry. Keep null and route it, or add the document.
The document contradicts itselfDo not retry. Set conflict_detected and route it.
Takeaway
  • A record that parses can still be wrong: semantic errors need your validator, not a stricter schema.
  • A retry without the specific errors is a fresh guess, and one without the document has nothing to re-read.
  • detected_pattern turns a dismissal rate into the name of the criterion to rewrite.

check yourself

0 / 3 answered

Your contract extractor returns records where end_date precedes effective_date, and the validator rejects them with "end_date 2025-03-01 is before effective_date 2027-03-01". In about half of these records one of the two dates was also misread, so the correct value has to be read from the contract again. Each retry is a new, stateless request.Which follow-up request most directly lets the model correct these records?

Select one.

Your support-email extractor fails validation on 9% of emails. Two causes dominate: request_date comes back as "3 March 2026" instead of ISO 8601, and order_id is empty when the customer writes that the order number is in an attached receipt. Attachments are never passed to the extractor. Every failure is retried three times with "invalid output, try again".Which TWO changes to the retry logic best cut wasted requests while keeping failures a retry could fix out of human review?

Select two.

The CI review bot posts about 60 findings a week, and developers dismiss 35% of them. Each finding carries file, line, severity and issue text, and nothing else. The team wants to rewrite the criteria behind the dismissals, deciding from the dismissal data itself and re-checking that data after each change.What is the most effective first step?

Select one.

sources

  1. Claude Platform Docs, "Structured outputs": "No retries needed for schema violations", the "Invalid outputs" cases (stop_reason "refusal" and "max_tokens", enum casing), and client.messages.parse() validating the response against a Pydantic model. Checked 2026-10-01.
  2. Claude Platform Docs, "Working with messages": the Messages API is stateless, so every request sends the full conversation history. Checked 2026-10-01.
  3. Claude Platform Docs, "Build a tool-using agent", error handling: send the error back in a tool_result with is_error: true so Claude can retry with corrected input. Checked 2026-10-01.
  4. Claude Platform Docs, "Prompting best practices", "Chain complex prompts": the self-correction chain of generate, review against criteria, and refine, as separate API calls. Checked 2026-10-01.
  5. Claude Platform Docs, "Reduce hallucinations": give Claude permission to say it lacks the information, and validate critical output. Checked 2026-10-01.
  6. The retry-with-error-feedback contents, the limits of retry and the external-document example, detected_pattern and dismissal analysis, and the calculated_total / stated_total / conflict_detected fields: Claude Certified Architect — Foundations Exam Guide v1.0, Domain 4.4. Scenarios: the guide's Structured Data Extraction and Claude Code for Continuous Integration. Invoice numbers and figures are this course's examples.