Task statement 4.2 · Prompt engineering and structured output
Apply few-shot prompting to improve output consistency and quality
The extractor returns an empty citations list for about a third of research reports, and every one of them lists its sources in a bibliography at the end instead of inline. The prompt already says, in bold, to include bibliography entries, so the instruction is right and the output is still wrong.
share
The tempting fix is to write the instruction again, longer or in capitals. That fails because the model is not ignoring the rule: it does not recognise a bibliography as a case the rule covers, and more words never show it one. The other tempting fix, a dozen examples of reports that already work, costs tokens on every request and teaches nothing new.
KnowledgeWhat an example does that an instruction cannot
An instruction describes a case in words, so the model has to recognise each new document as an instance of that description. A few-shot example shows the case instead: a real input, the correct output and, ideally, the reason the output is right. That is why few-shot examples are the most effective technique for consistently formatted, actionable output when detailed instructions alone produce inconsistent results. Write the instruction first, and add examples when the instruction is right but the output still varies.
A targeted example is one chosen because the model gets that case wrong or finds it ambiguous, not because it is typical. When the example says why one answer beat a plausible alternative, the model can "generalize judgment to novel patterns rather than matching only pre-specified cases", in the guide's words. For instance, a report with footnote citations is handled correctly after examples with inline citations and a bibliography, because the reasoning said that citations are any list of sources, wherever they appear. An example without its reasoning teaches only its layout. Every example also costs tokens on every request, so the guide asks for two to four targeted ones.
The same mechanism reduces hallucination in extraction across varied document structures. A required field that comes back empty or null usually means the value is there, but in a layout the model did not recognise: sources in a bibliography rather than inline citations, or a sample size embedded in a results sentence rather than stated in a methodology section. Examples from documents with varied formats show where the value lives in each one. The opposite failure is an invented value, and an example whose correct output is null shows that absence is an acceptable answer. The diagram runs three reports through the extractor with and without targeted examples.
Interactive · the same three documents, before and after examples
Detailed instructions only. Report A matches the description. Report B's layout is not recognised, and Report C gets a plausible number because nothing showed that null is allowed.
Instructions plus three targeted examples. Each is a case that failed, shown with why its answer is right, including an informal measurement whose correct output is null.
null. Report A needed no example, because its layout already worked.SkillWorked example: turning three failures into targeted examples
Report B came back with no citations and a null sample size. A first example that showed only the answer fixed Report B and nothing else. The second version adds one line of reasoning.
<example>
Input: [31-entry bibliography, no
inline cites; abstract: "over 1,000";
results: "1,208 households"]
Output: {"citations": [31 entries],
"sample_size": 1208}
</example>
// correct output, but no reason given,
// so a new layout is still a guess<example>
Input: [same report]
Reasoning: citations are any list of
sources, wherever they appear. Use
the exact 1,208 from the results,
not the rounded "over 1,000".
Output: {"citations": [31 entries],
"sample_size": 1208}
</example>The reasoning also settles an ambiguous case: two figures could be the sample size, and it says why the exact one beats the plausible alternative. Two more targeted examples come from cases the extractor got wrong. The first covers handling informal measurements: "we interviewed a few dozen farmers" came back as sample_size: 36, a value that parses, validates and is invented. The second is tool selection for ambiguous requests, because a "Final report and statement of costs" fits both extraction tools.
<example>
Input: [report: "we interviewed a few dozen farmers"]
Output: {"sample_size": null,
"sample_size_text": "a few dozen farmers"}
Reasoning: an informal measurement is not a count.
Keep the words; never convert them to a number.
</example>
<example>
Input: [12 pages of findings, then "Amount due: $18,400"]
Tool: extract_invoice
Reasoning: it requests payment. extract_report fits
the title but would drop the amount due.
</example>
That makes three examples, no two sharing a layout, and none showing Report A, which already worked. Each sits in <example> tags so the model can tell demonstration from instruction.
The bot's findings arrive in different shapes, some with no line number and some with no fix, although the prompt lists the four required parts. An example in the desired format (location, issue, severity, suggested fix) makes every finding take that shape. Examples that distinguish acceptable code patterns from genuine issues cut false positives while still letting the model generalise, because the reasoning names what makes a pattern safe. A test that calls apply_discount("SAVE10") makes the function look covered, but if no test reaches its expired-code branch, that is one of the branch-level test coverage gaps the bot should report.
<example> // genuine issue, in the required format Code: except Exception: pass # around charge_card() Finding: {"location": "billing/charge.py:88", "issue": "payment failure swallowed; order marked paid", "severity": "high", "suggested_fix": "catch GatewayError, log it, re-raise"} </example> <example> // acceptable pattern Code: except KeyError: return DEFAULT_TIMEOUT Finding: none Reasoning: a narrow catch with a deliberate fallback. Report catches that hide failures, not every try. </example>
More precise than the guide. Anthropic's prompting guide says "Include 3–5 examples for best results", where the exam guide says 2–4 (checked 2026-10-01). The docs ask for diverse examples and the guide for targeted ones, and both mean examples that differ from each other in the ways real inputs differ. Exam answer: two to four targeted examples.
- Few-shot examples fix inconsistent format and ambiguous judgement that a correct instruction could not fix alone.
- The reasoning line ("why this answer, not the plausible one") is what lets the model handle layouts and code it has never seen.
- In extraction, show each layout where a required field hides, and one case whose right answer is
null.