Task statement 4.5 · Prompt engineering and structured output
Design efficient batch processing strategies
The extraction system has two workloads. Suppliers upload an invoice and wait on the page while it is checked. Separately, about ten thousand scanned invoices and contracts a day must be in the ledger within 30 hours of arriving. Your manager proposes moving both to the Message Batches API to halve the cost.
share
The 50% saving makes batch look like a free win, so the tempting answer moves every workload onto it. That fails wherever someone is waiting, because a batch promises nothing shorter than 24 hours, and a supplier on the upload page cannot wait that long. The second tempting move is to resend the whole batch when a few requests fail, which pays again for every request that already succeeded and repeats the failures nobody fixed.
Start hereThree terms this section adds
- custom_id
- Your identifier on each request in a batch, such as
inv-20260921-00417. Results come back in any order, so it is how you match each result to its document. - latency-tolerant
- A non-blocking, latency-tolerant workload is one where nothing stops while it runs, so a result tomorrow morning is as good as one in a minute.
- blocking workflow
- A person or a pipeline cannot continue until the result arrives, such as a supplier on an upload page or a pre-merge check.
KnowledgeHow a batch runs
The Message Batches API takes many Messages requests in one call, each with a custom_id and the same params a normal Messages call takes. The requests are processed asynchronously, and all usage is charged at 50% of standard prices. Most batches finish in under an hour, but results become available only when every request has finished or 24 hours have passed, and requests still unprocessed at 24 hours expire. "Usually within an hour" is therefore not a design number, because the API guarantees no latency shorter than the 24-hour processing window.
- You create the batch, and its
processing_statusisin_progress. - You poll until the status becomes
ended. - You download the results, which stay available for 29 days.
- You match each result by
custom_id, because the order is not guaranteed. - You resubmit only the requests that did not succeed.
Choosing the API is a question about who waits. The supplier's upload check is a blocking workflow: the supplier stays on the page, so the answer must arrive in seconds, which only the synchronous Messages API does. The nightly ledger load is non-blocking and latency-tolerant, so a batch that takes eight hours costs nothing and saves half the price. Polling every minute does not make a batch finish sooner, and a fallback to the synchronous API after a timeout means you build both paths and still end up waiting on the synchronous one.
Synchronous Messages API
The supplier's upload check.
Result: full price, answer in seconds.
Message Batches API
Ten thousand scanned documents a day, 30-hour SLA.
Result: half price, results within the SLA.
SkillFitting 24 hours inside an SLA
When you submit a batch every few hours, the worst case is a document that arrives just after a submission. It waits a full interval for the next batch, and that batch may take the full 24 hours, so the worst case is the interval plus 24 hours. For the guide's 30-hour SLA, a submission every 4 hours gives a worst case of 28 hours and leaves 2 hours to validate and resubmit. Every 6 hours reaches exactly 30 hours, which meets the SLA only if nothing needs a second attempt.
Interactive · fitting a 24-hour window inside a 30-hour SLA
Choose how often a batch is submitted, and see the worst case for a document that arrives just after a submission.
Every 2 hours: worst case 2 + 24 = 26 hours. Four hours inside the SLA, at the cost of many small batches.
Every 4 hours: worst case 4 + 24 = 28 hours. Inside the 30-hour SLA with two hours to handle results. This is the guide's example.
Every 6 hours: worst case 6 + 24 = 30 hours. It just meets the SLA, with no time left to validate results or resubmit a failure.
Every 12 hours: worst case 12 + 24 = 36 hours. A document arriving just after a submission misses the SLA by six hours.
KnowledgeOne request, one turn of your tools
Each request in a batch runs on its own, with none of your code listening. If Claude answers with a tool_use block for a tool you run, that request ends there, because running the tool and sending back a tool_result takes a new request. So the batch API does not support multi-turn tool calling within a single request: it cannot execute your tools mid-request and return results. A single-turn extraction fits a batch well, while an agent loop over your own tools belongs on the synchronous API.
More precise than the guide. A batch request cannot stop, wait for your code to run a tool, and continue. The docs still list tool use and multi-turn conversations as batchable, because a request can carry earlier turns and their tool results, and server tools such as web search run inside it (checked 2026-10-01). Exam answer: the batch API does not support multi-turn tool calling within a single request.
SkillResubmitting only what failed
Once a batch has ended, each request is succeeded, errored, expired or canceled, and only succeeded is billed. A batch of 10,000 invoices ends with 9,962 succeeded and 38 errored. Resending all 10,000 pays again for 9,962 results you already have, and 31 of the 38 fail again: their invalid_request_error says each document exceeded the context limit, and an invalid request must be changed before it is resent. The fix is to resubmit only the 38, found by custom_id, with the oversized documents chunked under new IDs and the 7 server errors resent unchanged. Expired requests are resubmitted too.
# one request per document {"custom_id": "inv-20260921-00417", "params": {…, "tools": [extract_invoice], "messages": [{"role": "user", "content": "…document text…"}]}} # after the batch ended: 9,962 succeeded, 38 errored 31 × invalid_request_error — document exceeded the context limit → chunk into "inv-20260921-00417-p1" … "-p3", resubmit 7 × api_error — server error → resubmit unchanged, same custom_id
SkillRefine the prompt on a sample first
A batch reports its problems only after it ends, because even each request's params are validated asynchronously. A weak prompt therefore surfaces only after every document has cost money. Before batching the 10,000, run a sample of 50 documents that spans invoices, contracts and support emails through the synchronous API, and refine the prompt on what fails. Suppose 6 of the 50 put the PO number in invoice_number: one prompt fix and a second sample run take minutes, and the first-pass success rate rises before any batch is sent. Found after the batch, the same 12% error rate means about 1,200 resubmissions and another wait of up to 24 hours.
- Batch halves the price but promises only 24 hours, so it suits overnight reports, weekly audits and nightly test generation, never a pre-merge check.
- Every 6 hours just meets a 30-hour SLA with no slack. The guide's answer is every 4 hours, which leaves 2 hours to handle results.
- Match results by
custom_id, and resubmit only the failures, changing any request that was invalid.