Task statement 2.3 · Output evaluation and validation

2.3 · Fact-Checking and Validation

In Wrenfield Housing, the course example, the operations team asks Claude to summarise the housing regulator's new complaints guidance for the board. The summary reads well and cites paragraph numbers and deadlines with confidence. The board will plan its work around those details, so each one has to be checked against the guidance itself before the operations director signs the summary off.

Why it's tested

The tempting check is to ask Claude whether it is sure, or to read the summary again and see whether it sounds right. Both feel like diligence, but neither one looks at the guidance. Claude's answer to "are you sure?" comes from the same model that wrote the summary, with no new access to the source, so a wrong date can come back as confidently as a right one. The opposite habit fails too, because re-checking every sentence takes as long as reading the guidance, so the check gets skipped when time is short.

KnowledgeWhat a fact-check confirms

Fact-checking means confirming each specific claim in an output against an authoritative original: the document, dataset or official page the claim comes from, read in its own words. A specific claim is one that can be wrong to the letter, such as a number, a date, a citation, a quote or a name. These are the claims to check first, because readers act on them and because a hallucination, a confident detail with no basis in the source (2.2), often takes this form. Anthropic's help article on incorrect responses warns that Claude can show quotes that look authoritative but are not grounded in fact. It also says nobody should rely on Claude as a singular source of truth. This rules out any answer where the check is made against Claude, its tone or its own rating, instead of against the original.

Validation is the wider check that the output as a whole holds up: its figures recompute, its quotes exist in the source, and its claims do not contradict each other. Fact-checking works claim by claim, while validation asks whether the whole output is fit to send. A practical routine combines both, in this order:

  1. List the specific claims: every number, date, citation, quote and name the reader will act on.
  2. Ask Claude for its source: the exact sentence, page or paragraph behind each claim.
  3. Open the original: find that passage yourself and read the claim there, not in Claude's quote.
  4. Recompute the figures: redo each total or percentage from the source data.
  5. Mark each claim: confirmed, corrected, or removed when no source supports it.

Interactive · each claim in the board summary, checked against the original

Four specific claims from Claude's summary of the regulator's guidance, checked from top to bottom. On the left is each claim; the arrow names the check; on the right is the verdict. The paragraph citation is confirmed by opening the guidance at paragraph 4.7. The deadline is rejected, because paragraph 4.7 says 31 March 2027, not 30 September 2027. The 86% figure is confirmed by recomputing 184 divided by 214. The missing size threshold is confirmed as a gap by searching the guidance. A claim not checked yet is dashed and labelled not checked yet; a rejected claim is dotted and labelled rejected. CLAIM IN CLAUDE'S SUMMARY VERDICT AFTER THE CHECK Citation Paragraph 4.7 sets the self-assessment duty OPEN THE PDF AT 4.7 CONFIRMED 4.7 covers the self-assessment Deadline Self-assessment due by 30 September 2027 READ THE DATE IN 4.7 REJECTED 4.7 says 31 March 2027 → CORRECTED BEFORE THE BOARD NOT CHECKED YET Figure 86% of complaints answered within 20 working days RECOMPUTE THE SUM CONFIRMED 184 ÷ 214 = 86% FROM THE COMPLAINTS SUMMARY NOT CHECKED YET Gap Size threshold: “not stated in the guidance” SEARCH THE PDF CONFIRMED GAP no threshold anywhere → AN OPEN QUESTION FOR THE BOARD NOT CHECKED YET Each specific claim is checked against the original, not against the summary.

Step through the four specific claims in the board summary and see what each check finds.

The citation: paragraph 4.7 of the guidance does set the self-assessment duty, so the citation is real and on topic.

The deadline: paragraph 4.7 says 31 March 2027, not 30 September 2027. The paragraph was right and the date in it was wrong.

The figure: 184 of 214 complaints is 86%, recomputed from the complaints summary, so the sum holds.

The gap: Claude wrote “not stated” for a size threshold, and a search finds none, so the board sees an open question.

Dashed and labelled NOT CHECKED YET: a claim still to check. Dotted and labelled REJECTED: the original says something else.

KnowledgeTechniques, and what each one proves

Each technique proves something different, so the right one depends on the claim. Anthropic's guidance on reducing hallucinations lists three basic techniques that work in any chat: give Claude permission to say it does not know, ask it to ground its answer in direct quotes, and have it cite a source for each claim and withdraw any claim it cannot support. These make an answer checkable, but they do not check it. The same guidance says they reduce hallucinations without removing them, so critical information must still be validated. The checking itself is yours. For figures, the Claude Help Center says complex or mission-critical calculations should be verified with specialised software or by hand. Its web search article asks you to cross-reference cited sources for critical decisions. A second source counts only when it is independent, meaning it comes from neither Claude nor the document being checked:

Trace to the original

Find the paragraph, page or table the claim names, and read the claim there.

ProvesThe source says this, in these words.

Ask for quotes, then open them

"Quote the exact sentence behind each date and paragraph number."

ProvesNothing yet: the quote shows you where to look.

Recompute

Redo each total or percentage from the source figures, in a spreadsheet or by hand.

ProvesThe sum is right, though not that its inputs are.

Cross-check a second source

Compare with an independent source, such as the publisher's own announcement.

ProvesThe claim does not rest on one reading.

Allow "I don't know"

"If the guidance does not say, write 'not stated' rather than guessing."

ProvesA gap shows up as a gap, not as a guess.

Self-reported confidence is not on this list. When you ask Claude to rate how sure it is, the rating is more text from the same model, written without opening the source again, so it cannot tell a right date from a wrong one. The exam guide makes the same point: self-reported confidence is not a reliable accuracy signal. Rewording a summary to sound more formal or more cautious has the same flaw, because it changes how a claim sounds and leaves its truth unchecked.

SkillWorked example: the regulator's deadline

Wrenfield's operations team uploaded the housing regulator's new complaints guidance, a 38-page PDF, and asked Claude for a one-page summary for the December board meeting. The summary was clear and confident, and it said that paragraph 4.7 requires a self-assessment by 30 September 2027. The first check asked Claude to confirm its own work:

# before
Are you sure about paragraph 4.7 and the deadline?
→ "Yes. Paragraph 4.7 sets a deadline of 30 September 2027."
# after
For each paragraph number, date and figure in your summary,
quote the guidance's exact sentence. If the guidance does not
state something, write "not stated".
→ PDF opened at 4.7: "by 31 March 2027"; 184 ÷ 214 = 86% holds

The first check failed because it asked the summary's author to mark its own work, so the wrong date came back with more confidence, not less. The paragraph number was right, because 4.7 does set the self-assessment duty, but the deadline was six months too late. Claude's quoted sentence repeated the wrong date, which is why the team read paragraph 4.7 in the PDF instead of trusting the quote. The 86% figure recomputed correctly from the complaints summary, and Claude's "not stated" for a size threshold went to the board as an open question. Deciding when a draft needs more review, and by whom, is the next objective (2.4).

A real citation can carry a wrong fact. Paragraph 4.7 existed and covered the right topic, so checking only that the citation was real would have passed the wrong deadline. Read the claim inside the cited passage, not just the passage's heading.

Ask one question: for this specific claim, can you point to where an authoritative original says it, or redo its sum yourself? Yes, and it matches → keep it. No, or it differs → correct it from the original, or remove it and mark it unconfirmed.
Takeaway
  • Check the specific claims first: numbers, dates, citations, quotes and names, each against the original it depends on.
  • A quote or citation from Claude tells you where to look; opening the original is the check.
  • Asking Claude how sure it is checks nothing, because the answer never consults the source.

check yourself

0 / 3 answered

A school business manager asks Claude to summarise new government guidance on staff pay. The summary states, with no hedging, that "section 12(3) requires pay awards to be back-dated to 1 September." The governors' finance committee will set next year's budget from it, and the official guidance is published online.What is the most appropriate action before the manager sends the summary to the committee?

Select one.

A marketing analyst's Claude-written summary for a sales deck says a trade association's annual report shows the pet-food market grew 7.4% in 2025, to £3.9 billion. Claude also gave a link, which opens the association's home page. The deck's figures must match the report exactly.Which TWO steps would most directly confirm or correct both figures before the deck is shared?

Select two.

A grants officer uploads a funder's 30-page eligibility rules and asks Claude for the deadline, the maximum grant and the match-funding rule. Claude gives all three, but the rules never mention match funding, so Claude's 20% figure came from elsewhere. The officer wants the prompt to stop Claude filling gaps, not only to catch them later.Which change to the prompt most directly achieves this?

Select one.

sources

  1. Claude Platform Docs, "Reduce hallucinations": allow Claude to say "I don't know"; use direct quotes for factual grounding; verify with citations, and retract a claim with no supporting quote; the techniques reduce hallucinations but don't eliminate them, so always validate critical information. Checked 2026-10-15.
  2. Claude Help Center, "Claude is providing incorrect or misleading responses. What's going on?": Claude can display quotes that look authoritative but are not grounded in fact; do not rely on Claude as a singular source of truth; review Claude's cited sources. Checked 2026-10-15.
  3. Claude Help Center, "Enable and use web search": responses include citations so you can verify sources; cross-reference cited sources; use authoritative sources for critical decisions. Checked 2026-10-15.
  4. Claude Help Center, "How does Claude handle mathematical equations and calculations?": complex or mission-critical calculations should be verified with specialised software or manual methods. Checked 2026-10-15.
  5. Claude Certified Associate – Foundations exam guide, the answer rationale for its Domain 2 sample: self-reported confidence is not a reliable accuracy signal. Checked 2026-10-15.
  6. Wrenfield's board summary, the regulator's guidance, its paragraph 4.7 and both dates are this course's example; no real regulator or document is meant.