Gains:
- Determines which workloads batch processing is suitable for
- Understands the cost/latency tradeoff between synchronous, asynchronous and batch processing
- Designs a robust batch workflow that matches custom_id to results
Most LLM integrations focus on “live” scenarios where a user is waiting for a response in front of a screen. But the majority of professional workloads are not actually live: tagging thousands of documents overnight, summarizing an entire dataset, classifying entire call recordings in the archive. In these matters, no one expects an instant answer; The important thing is to finish the job cheaply and reliably. Batch is exactly for these workloads. In this unit, you'll learn the difference between synchronous, asynchronous, and batch processing, when batch is the right choice, and a robust flow that confidently matches custom_id and results.
Three Working Modes
mode
How does it work
delay
Typical cost
suitable job
synchronous
You make a request and wait for the response
seconds
Standard
Live chat, instant assistant
asynchronous
You queue the job and get notified when it's finished.
Seconds–minutes
Standard
Background tasks, automation steps
Batch
Sends thousands of requests in one package, then gets the results
Minutes–hours
Usually discounted
High-volume, delay-tolerant jobs
Batch processing is this: you send hundreds/thousands of requests as a single "job" to the provider; The provider processes them at its own pace and returns all results in bulk once completed. In return you get two things: (1) generally lower unit cost, (2) the ability to move high volume without having to deal with speed limits. The price is that the results do not come instantly, but after some time.
When to Batch, When Not?
The decision comes down to one question: Is the user waiting for the result now?
- No, I can hold it → batch candidate. Night tagging, batch summarization, archive classification, data enrichment, evaluation (eval) execution.
- Yes, waiting on the screen → sync. Live chat, instant advice, help when filling out forms.
Tip: Two modes can coexist in the same product. User works synchronously in live chat; At night, you give all the conversations of that day to the batch for quality analysis. Separating "living need" from "collective need" is the first decision of architecture.
Anatomy of Robust Batch Flow
The most important technical rule of batch processing is result matching.
- Give each request a unique `custom_id`. This is your generated ID that identifies the request (e.g. invoice-2026-07-18-000431).
- Submit the job. All requests go in one package; each with its own custom_id.
- Poll the situation. You ask for status at intervals until the job is "done."
- Match the results with `custom_id`. Results may be returned in a different order than the submission order; so never match by position but by the custom_id each result carries.
- Check the type of each result. One request might succeed, one might fail, one might expire. Process based on success/failure.
{ "requests": [ { "custom_id": "invoice-000431", "params": { "model": "claude-haiku-4-5", "max_tokens": 128, "system": "Classify invoice. Return JSON only.", "messages": [{ "role": "user", "content": "{{invoice_text}}" }] } }, { "custom_id": "invoice-000432", "params": { "model": "claude-haiku-4-5", "max_tokens": 128, "system": "Classify invoice. Return JSON only.", "messages": [{ "role": "user", "content": "{{invoice_text_2}}" }] } } ]}
Caution: Matching results based on submission order is the number one mistake in batching. The queue is not preserved. Without custom_id you can't confidently know which result belongs to which document — wrong matching silently leads to wrong data.
Copiable Templates
# custom_id generation rule (unique and traceable)Format: <isture>-<date>-<sequence>. Example: request-20260718-000431Rule: never repeat in work; Embed the resource record ID in it.
# Batch job card (scheduling template)Job name: .............Number of records: .............Model: ............. (simple job → fast model)Max_tokens per request: .............Expected delivery time tolerance: ......... hoursResult matching key: custom_idIn case of error: retry / queue / report
# Single request prompt in batch (short and schematic)Classify this document. Just return this JSON, commenting:{"category":"...","urgency":"low|medium|high"}Document: """{{document}}"""
# Result processing pseudo-code for each result: if result.status == "success": record = find(custom_id) save(record, result.output) otherwise: add_to_fail(custom_id, result.error) # then try again
Weak prompt / Strong prompt (batch job design)
# WEAK (fragile design)Send 10,000 documents in order with the strong model, save the returned results in the order they arrive.
# STRONG (durable design) Send 10,000 documents in one batch with a fast model. Give each document a unique custom_id containing the source-record ID. Match the results with the custom_id; queue the failed ones and try again.Run in the night window; Delivery tolerance 6 hours.
Powerful version; It pre-defines model selection, matching key, error handling and timing. This is the difference in securely processing tens of thousands of records.
Three Mini Cases
Case 1 — Night tagging. An e-commerce team would sort 200,000 product reviews into sentiment tags. Live synchronous streaming was subject to speed limits and was costly. They carried the work into the night as a batch with a fast model; The unit cost dropped, the entire set was ready in the morning, and there were no speed limit problems.
Case 2 — Order confusion. A research team batch abstracted 5,000 articles, but wrote the results into files in the order they arrived. Because the results were returned in a different order, approximately 900 of the 5,000 abstracts were linked to the wrong article. They remapped it to custom_id; problem solved and this experience became permanent rule: "Always custom_id in batch."
Case 3 — Live standby in wrong mode. A support team attempted to give batch the live responses the user expected on the screen; Users abandoned because the results arrived minutes later. They moved the live job back to synchronization, leaving only the nightly quality analysis in the batch. Lesson: batch is not for live standby.
Common mistakes
- Matching results by position: Order is not preserved; Use custom_id.
- Transferring live job to batch: User cannot wait for minutes; batch is for delay tolerant jobs.
- Not handling error cases: Some requests may return failed/expired; Put it in a separate queue and try again.
- Strong model usage reflex in batch: Fast model + batch is the cheapest combination in simple jobs.
- Not making custom_id traceable: If no source record is embedded in the ID, it becomes difficult to link the result back.
- Forgetting to examine the situation: Expecting results before the job is finished; Check completion status.
Deeper: Monitoring Batch and Managing Partial Failure
The most mature aspect of batch processing is that it requires a different mindset than individual calls: a batch job is a "process", not an "event". Assuming that tens of thousands of requests will all succeed is fragile; Realistic design accepts partial failure from the start. The status of each result can be different: successful, failed (e.g. invalid input), canceled or expired. A robust flow processes the status of each result separately as it travels through it, puts the failures into a separate "retry queue" and runs that queue separately.
The second practice is to design for idempotency (that running the same job twice doesn't cause any harm). If a batch is interrupted and you restart it, you should not reprocess and write twice the already processed records. Binding the custom_id to your source record works here too: "has this record already been processed?" before saving the result. Checking prevents double typing.
The third point is to stagger live streams with batch. Some jobs have both live and batch dimensions: when the user loads a document, you give them a quick preliminary summary (synchronous), and reprocess the same document for deeper analysis at night (batch). Consciously separating the two modes optimizes both user experience and cost.
Finally, batching is also a way to deal with speed limits (unit 8). Sending high volume in live synchronous flow produces constant 429, while sending the same volume to batch transfers limit pressure to the provider's own scheduling and makes the job more predictable.
In summary
Batch processing is generally a cheaper and more robust mode for latency-tolerant and high-volume workloads. His decision was "is the user waiting for the result now?" determines the question. The most critical technical rule is to give each request a unique custom_id, match results by ID rather than location, and treat each result's success/failure separately.
Application task
Choose a high-volume job (e.g. archive classification). (1) Decide whether this work is live or collective and justify it. (2) Design a custom_id format (include the resource record). (3) Fill out the batch job card (model, max_tokens, tolerance, error policy). (4) Write the result processing pseudocode to include failed requests.
checklist
- [ ] I can distinguish synchronous, asynchronous and batch modes on the cost/delay axis.
- [ ] I can decide whether a job is suitable for batch or not by asking the right question.
- [ ] I give each request a unique custom_id and match the results by ID.
- [ ] I can handle failed/expired results separately.
- [ ] I know the benefits of choosing a fast model in simple batch jobs.