Task statement 2.3 · Output evaluation and validation
2.3 · Fact-Checking and Validation
In Wrenfield Housing, the course example, the operations team asks Claude to summarise the housing regulator's new complaints guidance for the board. The summary reads well and cites paragraph numbers and deadlines with confidence. The board will plan its work around those details, so each one has to be checked against the guidance itself before the operations director signs the summary off.
share
The tempting check is to ask Claude whether it is sure, or to read the summary again and see whether it sounds right. Both feel like diligence, but neither one looks at the guidance. Claude's answer to "are you sure?" comes from the same model that wrote the summary, with no new access to the source, so a wrong date can come back as confidently as a right one. The opposite habit fails too, because re-checking every sentence takes as long as reading the guidance, so the check gets skipped when time is short.
KnowledgeWhat a fact-check confirms
Fact-checking means confirming each specific claim in an output against an authoritative original: the document, dataset or official page the claim comes from, read in its own words. A specific claim is one that can be wrong to the letter, such as a number, a date, a citation, a quote or a name. These are the claims to check first, because readers act on them and because a hallucination, a confident detail with no basis in the source (2.2), often takes this form. Anthropic's help article on incorrect responses warns that Claude can show quotes that look authoritative but are not grounded in fact. It also says nobody should rely on Claude as a singular source of truth. This rules out any answer where the check is made against Claude, its tone or its own rating, instead of against the original.
Validation is the wider check that the output as a whole holds up: its figures recompute, its quotes exist in the source, and its claims do not contradict each other. Fact-checking works claim by claim, while validation asks whether the whole output is fit to send. A practical routine combines both, in this order:
- List the specific claims: every number, date, citation, quote and name the reader will act on.
- Ask Claude for its source: the exact sentence, page or paragraph behind each claim.
- Open the original: find that passage yourself and read the claim there, not in Claude's quote.
- Recompute the figures: redo each total or percentage from the source data.
- Mark each claim: confirmed, corrected, or removed when no source supports it.
Interactive · each claim in the board summary, checked against the original
Step through the four specific claims in the board summary and see what each check finds.
The citation: paragraph 4.7 of the guidance does set the self-assessment duty, so the citation is real and on topic.
The deadline: paragraph 4.7 says 31 March 2027, not 30 September 2027. The paragraph was right and the date in it was wrong.
The figure: 184 of 214 complaints is 86%, recomputed from the complaints summary, so the sum holds.
The gap: Claude wrote “not stated” for a size threshold, and a search finds none, so the board sees an open question.
KnowledgeTechniques, and what each one proves
Each technique proves something different, so the right one depends on the claim. Anthropic's guidance on reducing hallucinations lists three basic techniques that work in any chat: give Claude permission to say it does not know, ask it to ground its answer in direct quotes, and have it cite a source for each claim and withdraw any claim it cannot support. These make an answer checkable, but they do not check it. The same guidance says they reduce hallucinations without removing them, so critical information must still be validated. The checking itself is yours. For figures, the Claude Help Center says complex or mission-critical calculations should be verified with specialised software or by hand. Its web search article asks you to cross-reference cited sources for critical decisions. A second source counts only when it is independent, meaning it comes from neither Claude nor the document being checked:
Find the paragraph, page or table the claim names, and read the claim there.
ProvesThe source says this, in these words.
"Quote the exact sentence behind each date and paragraph number."
ProvesNothing yet: the quote shows you where to look.
Redo each total or percentage from the source figures, in a spreadsheet or by hand.
ProvesThe sum is right, though not that its inputs are.
Compare with an independent source, such as the publisher's own announcement.
ProvesThe claim does not rest on one reading.
"If the guidance does not say, write 'not stated' rather than guessing."
ProvesA gap shows up as a gap, not as a guess.
Self-reported confidence is not on this list. When you ask Claude to rate how sure it is, the rating is more text from the same model, written without opening the source again, so it cannot tell a right date from a wrong one. The exam guide makes the same point: self-reported confidence is not a reliable accuracy signal. Rewording a summary to sound more formal or more cautious has the same flaw, because it changes how a claim sounds and leaves its truth unchecked.
SkillWorked example: the regulator's deadline
Wrenfield's operations team uploaded the housing regulator's new complaints guidance, a 38-page PDF, and asked Claude for a one-page summary for the December board meeting. The summary was clear and confident, and it said that paragraph 4.7 requires a self-assessment by 30 September 2027. The first check asked Claude to confirm its own work:
# before Are you sure about paragraph 4.7 and the deadline? → "Yes. Paragraph 4.7 sets a deadline of 30 September 2027." # after For each paragraph number, date and figure in your summary, quote the guidance's exact sentence. If the guidance does not state something, write "not stated". → PDF opened at 4.7: "by 31 March 2027"; 184 ÷ 214 = 86% holds
The first check failed because it asked the summary's author to mark its own work, so the wrong date came back with more confidence, not less. The paragraph number was right, because 4.7 does set the self-assessment duty, but the deadline was six months too late. Claude's quoted sentence repeated the wrong date, which is why the team read paragraph 4.7 in the PDF instead of trusting the quote. The 86% figure recomputed correctly from the complaints summary, and Claude's "not stated" for a size threshold went to the board as an open question. Deciding when a draft needs more review, and by whom, is the next objective (2.4).
A real citation can carry a wrong fact. Paragraph 4.7 existed and covered the right topic, so checking only that the citation was real would have passed the wrong deadline. Read the claim inside the cited passage, not just the passage's heading.
- Check the specific claims first: numbers, dates, citations, quotes and names, each against the original it depends on.
- A quote or citation from Claude tells you where to look; opening the original is the check.
- Asking Claude how sure it is checks nothing, because the answer never consults the source.