Task statement 4.2 · Prompt engineering and structured output

Apply few-shot prompting to improve output consistency and quality

The extractor returns an empty citations list for about a third of research reports, and every one of them lists its sources in a bibliography at the end instead of inline. The prompt already says, in bold, to include bibliography entries, so the instruction is right and the output is still wrong.

Why it's tested

The tempting fix is to write the instruction again, longer or in capitals. That fails because the model is not ignoring the rule: it does not recognise a bibliography as a case the rule covers, and more words never show it one. The other tempting fix, a dozen examples of reports that already work, costs tokens on every request and teaches nothing new.

KnowledgeWhat an example does that an instruction cannot

An instruction describes a case in words, so the model has to recognise each new document as an instance of that description. A few-shot example shows the case instead: a real input, the correct output and, ideally, the reason the output is right. That is why few-shot examples are the most effective technique for consistently formatted, actionable output when detailed instructions alone produce inconsistent results. Write the instruction first, and add examples when the instruction is right but the output still varies.

A targeted example is one chosen because the model gets that case wrong or finds it ambiguous, not because it is typical. When the example says why one answer beat a plausible alternative, the model can "generalize judgment to novel patterns rather than matching only pre-specified cases", in the guide's words. For instance, a report with footnote citations is handled correctly after examples with inline citations and a bibliography, because the reasoning said that citations are any list of sources, wherever they appear. An example without its reasoning teaches only its layout. Every example also costs tokens on every request, so the guide asks for two to four targeted ones.

The same mechanism reduces hallucination in extraction across varied document structures. A required field that comes back empty or null usually means the value is there, but in a layout the model did not recognise: sources in a bibliography rather than inline citations, or a sample size embedded in a results sentence rather than stated in a methodology section. Examples from documents with varied formats show where the value lives in each one. The opposite failure is an invented value, and an example whose correct output is null shows that absence is an acceptable answer. The diagram runs three reports through the extractor with and without targeted examples.

Interactive · the same three documents, before and after examples

Three research documents with different layouts flow down into extracted citations and sample size fields; with instructions only, two of the three extractions are wrong; with targeted examples, all three are correct THREE INCOMING REPORTS · DIFFERENT LAYOUTS Report A Citations inline: "(Okafor, 2021)" Methodology section: "n = 412 participants" THE LAYOUT THE PROMPT EXPECTS Report B 31 numbered entries in a bibliography at the end "…of the 1,208 households surveyed…" SIZE EMBEDDED IN A RESULTS SENTENCE Report C Inline citations, a literature review. No sample is described anywhere in the document. THE FIELD IS ABSENT extract extract extract WHAT COMES BACK citations: 14 entries sample_size: 412 CORRECT citations: [ ] citations: 31 entries sample_size: null sample_size: 1208 BOTH FIELDS MISSED CORRECT citations: 9 entries sample_size: 200 sample_size: null SAMPLE SIZE INVENTED CORRECT · NULL IS THE ANSWER Instructions only: the layout the prompt was written around works. With targeted examples: a bibliography, a figure in running text and a null answer The other two layouts each fail differently, and rewording the rule does not reach either. have each been shown once with the reasoning, and all three reports come back right.

Detailed instructions only. Report A matches the description. Report B's layout is not recognised, and Report C gets a plausible number because nothing showed that null is allowed.

Instructions plus three targeted examples. Each is a case that failed, shown with why its answer is right, including an informal measurement whose correct output is null.

Illustrative run. Time reads downward: document, extraction, record. The second state adds the targeted examples from the worked example below: a bibliography report whose sample size sits in running text, and an informal measurement whose correct output is null. Report A needed no example, because its layout already worked.

SkillWorked example: turning three failures into targeted examples

Report B came back with no citations and a null sample size. A first example that showed only the answer fixed Report B and nothing else. The second version adds one line of reasoning.

Shows the answer
<example>
Input: [31-entry bibliography, no
  inline cites; abstract: "over 1,000";
  results: "1,208 households"]
Output: {"citations": [31 entries],
         "sample_size": 1208}
</example>
// correct output, but no reason given,
// so a new layout is still a guess
Shows the answer and the reasoning
<example>
Input: [same report]
Reasoning: citations are any list of
  sources, wherever they appear. Use
  the exact 1,208 from the results,
  not the rounded "over 1,000".
Output: {"citations": [31 entries],
         "sample_size": 1208}
</example>

The reasoning also settles an ambiguous case: two figures could be the sample size, and it says why the exact one beats the plausible alternative. Two more targeted examples come from cases the extractor got wrong. The first covers handling informal measurements: "we interviewed a few dozen farmers" came back as sample_size: 36, a value that parses, validates and is invented. The second is tool selection for ambiguous requests, because a "Final report and statement of costs" fits both extraction tools.

<example>
Input: [report: "we interviewed a few dozen farmers"]
Output: {"sample_size": null,
         "sample_size_text": "a few dozen farmers"}
Reasoning: an informal measurement is not a count.
  Keep the words; never convert them to a number.
</example>
<example>
Input: [12 pages of findings, then "Amount due: $18,400"]
Tool: extract_invoice
Reasoning: it requests payment. extract_report fits
  the title but would drop the amount due.
</example>

That makes three examples, no two sharing a layout, and none showing Report A, which already worked. Each sits in <example> tags so the model can tell demonstration from instruction.

Scenario switchThis part uses the domain's second scenario, Claude Code for Continuous Integration. The guide writes three of its few-shot skills about code review, so the pull request review bot from 4.1 is the system here.

The bot's findings arrive in different shapes, some with no line number and some with no fix, although the prompt lists the four required parts. An example in the desired format (location, issue, severity, suggested fix) makes every finding take that shape. Examples that distinguish acceptable code patterns from genuine issues cut false positives while still letting the model generalise, because the reasoning names what makes a pattern safe. A test that calls apply_discount("SAVE10") makes the function look covered, but if no test reaches its expired-code branch, that is one of the branch-level test coverage gaps the bot should report.

<example> // genuine issue, in the required format
Code: except Exception: pass   # around charge_card()
Finding: {"location": "billing/charge.py:88",
  "issue": "payment failure swallowed; order marked paid",
  "severity": "high",
  "suggested_fix": "catch GatewayError, log it, re-raise"}
</example>
<example> // acceptable pattern
Code: except KeyError: return DEFAULT_TIMEOUT
Finding: none
Reasoning: a narrow catch with a deliberate fallback.
  Report catches that hide failures, not every try.
</example>

More precise than the guide. Anthropic's prompting guide says "Include 3–5 examples for best results", where the exam guide says 2–4 (checked 2026-10-01). The docs ask for diverse examples and the guide for targeted ones, and both mean examples that differ from each other in the ways real inputs differ. Exam answer: two to four targeted examples.

Ask one question: is the instruction already right, and is the output still inconsistent or wrong on particular cases? Yes → add two to four targeted examples of those cases, each with the reasoning for its answer. No → fix the instruction or tool description first, because examples cannot rescue a wrong one.
Takeaway
  • Few-shot examples fix inconsistent format and ambiguous judgement that a correct instruction could not fix alone.
  • The reasoning line ("why this answer, not the plausible one") is what lets the model handle layouts and code it has never seen.
  • In extraction, show each layout where a required field hides, and one case whose right answer is null.

check yourself

0 / 3 answered

The extractor fills requested_action (refund, replacement, information or unclear) from support emails. Emails that both complain and ask a question get different values on different runs. The prompt already defines each value in a paragraph. The fix must also cover mixed emails phrased in ways the team has not collected yet.Which change most effectively makes these labels consistent?

Select one.

The CI review bot flags every hardcoded password, including dummy values such as password = "test123" in test fixtures, and developers now dismiss credential findings unread. Last month a real production key was committed inside tests/, and the team wants the bot to catch that case next time.Which TWO changes best cut these false positives while still catching real secrets?

Select two.

You can afford three few-shot examples in the invoice extractor's prompt, which already describes every case below. Error analysis of 200 invoices: 120 standard layouts, all correct; 30 multi-currency invoices with wrong totals; 25 credit notes sent to extract_invoice; 25 with quantities like "approx. 2 crates" turned into exact numbers.Which set of three examples most improves overall accuracy?

Select one.

sources

  1. Claude Platform Docs, "Prompting best practices" — "Use examples effectively": examples as one of the most reliable ways to steer format, tone and structure; the relevant / diverse / structured criteria; <example> tags; the 3–5 recommendation (checked 2026-10-01).
  2. Claude Platform Docs, "Reduce hallucinations" — "Allow Claude to say 'I don't know'", which a null example demonstrates (checked 2026-10-01).
  3. Task statement text, the 2–4 targeted-examples figure, and the citation, methodology, informal-measurement, tool-selection, test-coverage and code-review examples: Claude Certified Architect — Foundations Exam Guide v1.0, Domain 4.2.