Task statement 7.1 · Troubleshooting and optimization
7.1 · Diagnosing Poor Outputs
In Wrenfield Housing, the course example, housing officers share one saved prompt that turns anonymised estate-inspection notes into an action list for each estate. The lists keep coming back wrong in more than one way, and each officer who patches the prompt guesses differently about why.
share
The tempting response to a poor output is to change several things at once: rewrite the prompt, add "be thorough", or switch to a bigger model. One change may help, but you cannot tell which, and a change aimed at the wrong cause can break a part that worked. Switching the model needs no thought about the prompt, yet most poor outputs trace to something the prompt or its material left open, so a different model still has to guess.
KnowledgeChoose the cause before you change anything
A prompt is underperforming when its results keep failing the checks you set for them. To diagnose it, you work back from the symptom (what is wrong in the output) to the cause (what in the prompt or its material let that happen). To resolve it, you make the one change that cause needs and show that it worked. The same symptom can have different causes: a generic answer may come from a missing audience or a missing example, and only the prompt tells you which. This rules out changing the prompt, the model or the feature before the cause is named.
Diagnosis then runs in four steps, and it repeats for each symptom:
- Name the symptom: quote what is wrong, such as a missing item or an invented figure.
- Choose the likely cause: point to the line, or the missing part, that let it happen.
- Change one thing: the fix that cause needs, and nothing else.
- Re-test on the same inputs: run it on the material that failed, against the same checks (1.3).
Interactive · three symptoms, three causes, one change each
Step through the three changes the officers made to the shared prompt, one symptom at a time.
Change 1 removes a request the source cannot answer. The re-run shows no invented priorities.
Change 2 splits one step into two: list, then write. The re-run covers all 23 defects.
Change 3 adds one example from a past list. Every action on the re-run names a place and an owner.
Anthropic's prompting advice is to test each addition to see whether it actually improves results, and that works only when one change separates a run from the last one. The same inputs matter because new material brings its own differences, so a better result on easier notes proves nothing. When results differ from run to run, run the same input twice, both before and after the change.
KnowledgeA map from symptom to cause
Most poor outputs fall into a few groups, each pointing to a few causes. Anthropic's troubleshooting list pairs them too: too generic needs specificity or examples, an inconsistent format needs examples, and made-up information needs permission to say "I don't know". In the map, stale project knowledge means an out-of-date file in a Project, which repeats its error in every chat there until someone replaces it. Each fix is a habit from earlier in the course; the new skill is choosing the right one:
Bland text that could suit any reader; an answer to a different question.
Likely cause → fixMissing context or audience, or an unclear task → add the missing part of the prompt (1.1).
The wrong format or length; a different layout on each run.
Likely cause → fixNo example, so the shape is left open → add one example and the length (1.1).
Missing items or a section; an instruction ignored; drift late in a long chat.
Likely cause → fixToo much in one step → split it (1.2). A buried or conflicting instruction → short instructions with reasons (5.3). A long chat → restart with only what is needed (3.4).
Invented details; facts that were right last year.
Likely cause → fixA request beyond the source → ask only for what it holds and allow "not stated" (2.3). Stale project knowledge → replace the file (5.4).
A colleague could do the task from the prompt (1.1), yet results still fall short.
Likely cause → fixThe wrong feature or model → change it after testing real examples (3.1, 3.3). A task that should not go to Claude → keep it with people (6.1).
No rewrite rescues a task sent to the wrong place, so the last group needs checking whenever a clear prompt fails. A hallucination often follows a request beyond the source. When asked for something the material does not hold, Claude tends to supply a plausible value rather than leave a gap. Anthropic's guidance on reducing hallucinations starts with explicit permission to say it does not know.
SkillWorked example: the inspection action lists
Wrenfield's housing officers inspect each of its 11 estates every quarter. Their notes name blocks and landings but no tenants, as the AI policy requires. The officers paste the notes under a shared prompt kept in the operations team's folder. After the spring round, three faults stood out, as the figure shows: an invented priority, 6 of Millbrook's 23 defects missing, and generic actions. Before changing anything, the team wrote three checks and fixed a test set: the spring notes for Millbrook (12 pages), Ashgrove (3 pages) and Fenmoor (5 pages).
# before Turn these inspection notes into an action list for the estate, with a priority for each action. → Millbrook: 17 actions for 23 defects; one invented priority # after three changes, one per re-test <notes> … </notes> Step 1: list every defect in the notes, with its page. Step 2: turn each defect into one action, like this example: "Block C, 2nd-floor landing: replace the broken light. Owner: caretaker." Priority: write "officer to set" unless the notes give one. → all three estates: 3 of 3 checks pass
The team fixed the invented priority first, because a false fact misleads the people acting on the list, while a vague action only slows them down. The prompt asked for a priority the notes never hold, so the change let the officer set it. The missed defects traced to one step doing too much on long notes, and the generic actions to a missing example. Each change was re-run on the same notes against the same checks: every defect has an action, each action says what, where and who, and nothing appears that the notes do not say. The officers still read every list, and a dated note records what changed.
When a change does not help. If the re-test does not move, the cause was probably wrong, so go back to step 2 rather than piling on changes. If the fix needs another system or an integration, escalate it to a developer or architect.
- Find the cause before the fix, because one symptom can have several causes, each with its own fix.
- One change per re-test, on the same inputs, so any improvement traces to its change.
- A clear prompt that still fails points to the wrong feature or model, or a task that should stay with people.