Task statement 7.1 · Troubleshooting and optimization

7.1 · Diagnosing Poor Outputs

In Wrenfield Housing, the course example, housing officers share one saved prompt that turns anonymised estate-inspection notes into an action list for each estate. The lists keep coming back wrong in more than one way, and each officer who patches the prompt guesses differently about why.

Why it's tested

The tempting response to a poor output is to change several things at once: rewrite the prompt, add "be thorough", or switch to a bigger model. One change may help, but you cannot tell which, and a change aimed at the wrong cause can break a part that worked. Switching the model needs no thought about the prompt, yet most poor outputs trace to something the prompt or its material left open, so a different model still has to guess.

KnowledgeChoose the cause before you change anything

A prompt is underperforming when its results keep failing the checks you set for them. To diagnose it, you work back from the symptom (what is wrong in the output) to the cause (what in the prompt or its material let that happen). To resolve it, you make the one change that cause needs and show that it worked. The same symptom can have different causes: a generic answer may come from a missing audience or a missing example, and only the prompt tells you which. This rules out changing the prompt, the model or the feature before the cause is named.

Diagnosis then runs in four steps, and it repeats for each symptom:

  1. Name the symptom: quote what is wrong, such as a missing item or an invented figure.
  2. Choose the likely cause: point to the line, or the missing part, that let it happen.
  3. Change one thing: the fix that cause needs, and nothing else.
  4. Re-test on the same inputs: run it on the material that failed, against the same checks (1.3).

Interactive · three symptoms, three causes, one change each

Three columns, one for each symptom in the inspection action lists, each read from top to bottom. In each column the symptom is traced to a cause, the cause needs one change, and the changed prompt is re-run on the same three estates' notes, which gives the re-test result. Column 1: an invented priority, traced to a prompt that asked for priorities the notes do not hold; the change lets the officer set the priority; no invented priorities remain. Column 2: six defects missed from twelve pages of notes, traced to doing everything in one step; the change lists every defect first; 23 of 23 defects get an action. Column 3: generic actions, traced to no example of a good action; the change adds one example; every action says what, where and who. A column not being changed is dashed and labelled not this change. CHANGE 1 · INVENTED PRIORITY NOT THIS CHANGE Symptom “High: within 48 hours” for a broken bin store door TRACED TO Cause asked for a priority, but the notes hold none NEEDS One change “Priority: officer to set”, unless the notes give one RE-RUN, SAME NOTES Re-test no invented priorities CHANGE 2 · MISSED DEFECTS NOT THIS CHANGE Symptom Millbrook: 6 of 23 defects missed, all on pages 9–12 TRACED TO Cause 12 pages read and turned into actions in one step NEEDS One change first list every defect, then write the actions RE-RUN, SAME NOTES Re-test 23 of 23 defects listed CHANGE 3 · GENERIC ACTIONS NOT THIS CHANGE Symptom “Improve communal cleaning”: no where, no who TRACED TO Cause no example of a good action in the prompt NEEDS One change one example action: what, where, who, by when RE-RUN, SAME NOTES Re-test every action: what, where, who Each change answers one cause, and each re-test uses the same three estates’ notes.

Step through the three changes the officers made to the shared prompt, one symptom at a time.

Change 1 removes a request the source cannot answer. The re-run shows no invented priorities.

Change 2 splits one step into two: list, then write. The re-run covers all 23 defects.

Change 3 adds one example from a past list. Every action on the re-run names a place and an owner.

Dashed and labelled NOT THIS CHANGE: a symptom left alone for now.

Anthropic's prompting advice is to test each addition to see whether it actually improves results, and that works only when one change separates a run from the last one. The same inputs matter because new material brings its own differences, so a better result on easier notes proves nothing. When results differ from run to run, run the same input twice, both before and after the change.

KnowledgeA map from symptom to cause

Most poor outputs fall into a few groups, each pointing to a few causes. Anthropic's troubleshooting list pairs them too: too generic needs specificity or examples, an inconsistent format needs examples, and made-up information needs permission to say "I don't know". In the map, stale project knowledge means an out-of-date file in a Project, which repeats its error in every chat there until someone replaces it. Each fix is a habit from earlier in the course; the new skill is choosing the right one:

Generic or off the point

Bland text that could suit any reader; an answer to a different question.

Likely cause → fixMissing context or audience, or an unclear task → add the missing part of the prompt (1.1).

Wrong shape

The wrong format or length; a different layout on each run.

Likely cause → fixNo example, so the shape is left open → add one example and the length (1.1).

Left out or ignored

Missing items or a section; an instruction ignored; drift late in a long chat.

Likely cause → fixToo much in one step → split it (1.2). A buried or conflicting instruction → short instructions with reasons (5.3). A long chat → restart with only what is needed (3.4).

Not true

Invented details; facts that were right last year.

Likely cause → fixA request beyond the source → ask only for what it holds and allow "not stated" (2.3). Stale project knowledge → replace the file (5.4).

Poor after a clear prompt

A colleague could do the task from the prompt (1.1), yet results still fall short.

Likely cause → fixThe wrong feature or model → change it after testing real examples (3.1, 3.3). A task that should not go to Claude → keep it with people (6.1).

No rewrite rescues a task sent to the wrong place, so the last group needs checking whenever a clear prompt fails. A hallucination often follows a request beyond the source. When asked for something the material does not hold, Claude tends to supply a plausible value rather than leave a gap. Anthropic's guidance on reducing hallucinations starts with explicit permission to say it does not know.

SkillWorked example: the inspection action lists

Wrenfield's housing officers inspect each of its 11 estates every quarter. Their notes name blocks and landings but no tenants, as the AI policy requires. The officers paste the notes under a shared prompt kept in the operations team's folder. After the spring round, three faults stood out, as the figure shows: an invented priority, 6 of Millbrook's 23 defects missing, and generic actions. Before changing anything, the team wrote three checks and fixed a test set: the spring notes for Millbrook (12 pages), Ashgrove (3 pages) and Fenmoor (5 pages).

# before
Turn these inspection notes into an action list for the estate,
with a priority for each action.
→ Millbrook: 17 actions for 23 defects; one invented priority
# after three changes, one per re-test
<notes> … </notes>
Step 1: list every defect in the notes, with its page.
Step 2: turn each defect into one action, like this example:
"Block C, 2nd-floor landing: replace the broken light. Owner: caretaker."
Priority: write "officer to set" unless the notes give one.
→ all three estates: 3 of 3 checks pass

The team fixed the invented priority first, because a false fact misleads the people acting on the list, while a vague action only slows them down. The prompt asked for a priority the notes never hold, so the change let the officer set it. The missed defects traced to one step doing too much on long notes, and the generic actions to a missing example. Each change was re-run on the same notes against the same checks: every defect has an action, each action says what, where and who, and nothing appears that the notes do not say. The officers still read every list, and a dated note records what changed.

When a change does not help. If the re-test does not move, the cause was probably wrong, so go back to step 2 rather than piling on changes. If the fix needs another system or an integration, escalate it to a developer or architect.

Ask one question: can you point to what in the prompt or its material caused this symptom? Yes → change that one thing and re-run the same inputs. No → read the prompt against the map before you change anything.
Takeaway
  • Find the cause before the fix, because one symptom can have several causes, each with its own fix.
  • One change per re-test, on the same inputs, so any improvement traces to its change.
  • A clear prompt that still fails points to the wrong feature or model, or a task that should stay with people.

check yourself

0 / 3 answered

A marketing coordinator's saved prompt turns a week of customer survey comments into a one-page digest. This week's digest misses two of the five themes in the comments and calls one theme "trending upward", though the prompt only ever supplies one week of comments. The coordinator wants to know which change fixed which fault.What is the most effective first step?

Select one.

A finance assistant has drafted monthly expense summaries in one chat for six weeks, now about 120 messages long. In the first message they asked for every amount in euros with two decimal places, and the summaries followed it until this week. Today's summary rounds every amount and shows no currency. The data file has the same layout as before.What is the most likely root cause?

Select one.

An IT service desk uses a Project to answer staff questions from an uploaded policy file. Answers still give last year's password-reset steps, although the policy changed last month and the file was never replaced. Answers also switch between a paragraph and a numbered list, though staff need numbered steps. The desk lead will test any fix on last week's questions.Which TWO changes most directly fix the causes?

Select two.

sources

  1. Claude Blog, "Best practices for prompt engineering" (2025-11-10): the "Troubleshooting common prompt issues" list (too generic → specificity or examples; inconsistent format → examples; complex task with unreliable results → split into prompts that each do one thing; made-up information → permission to say "I don't know"); test each addition to see if it actually improves results. Checked 2026-10-15.
  2. Claude Platform Docs, "Reduce hallucinations": explicitly give Claude permission to admit uncertainty; restrict it to the provided documents; run the same prompt several times and compare the outputs to spot inconsistencies. Checked 2026-10-15.
  3. Claude Platform Docs, "Prompting best practices": examples are one of the most reliable ways to steer format, tone and structure, and improve consistency; put long documents near the top of the prompt, above the instructions. Checked 2026-10-15.
  4. Claude Help Center, "My prompt isn't giving me a helpful answer": give as much context as possible; break down complex requests into substeps; give Claude feedback. Checked 2026-10-15.
  5. Wrenfield's inspection notes, the shared prompt, its three faults and the re-test results are this course's example.