Gains:
- Can compare model family (fast/balanced/powerful) on capability, speed and cost
- Designs model selection and routing strategies according to task complexity
- Bases model selection on evidence with a small set of evals
The single decision that determines your most bang for your buck and quality in LLM integration is which model you use. The common reflex is “choose the strongest model”; However, this often means unnecessary costs and delays. The right approach is to choose the lightest model that accomplishes each task and base that choice on measurement, not guesswork. In this unit, you will compare the model family on the capability/speed/cost axis, establish a model routing strategy according to task complexity, and evidence the selection with a small set of evaluations.
Understanding the Model Family
Providers generally offer three classes: fast/cheap, stable and powerful. The relationship between them is summarized on three axes: ability (power to solve difficult tasks), speed (latency), cost (token price).
class
example
talent
speed
Cost
Available tasks
fast
Haiku 4.5
medium
very high
low
Classification, labeling, brief summary, orientation
balanced
sonnet 5
high
high
medium
General purpose, coding, multi-step flow, most agent work
strong
Opus 4.8
highest
medium
high
Complex reasoning, long-range autonomous tasks, difficult analysis
Critical insight: the more powerful model does not perform better on every job. In a simple "urgent or not" labeling, the strong model and the fast model give the same correct answer; the only difference is that the powerful is 5 times more expensive and slower. Extra talent produces value only when the mission requires it.
Step by Step: How to Choose a Model?
- Classify the task. Is it routine/patterned (labeling, inference), or open-ended/multi-step (analysis, planning, code)?
- Start with the lightest candidate. Try it with the fast model. If that's enough, stop.
- If it's not enough, move up to a higher class. If the accuracy is low, go to the balanced one, if that is not enough, go to the strong one.
- Measure, don't guess. Compare the accuracy and cost of each candidate with a small set of eval (below).
- Set up redirection. Instead of connecting to a single model, distribute the task to the right model with a "router".
Model Routing
Real workloads are mixed: most incoming requests are simple, some are difficult. It's a waste to send them all to the powerful model; Sending them all to the fast model reduces quality. Routing solves this: a cheap model (or a simple rule) classifies the task first, then the job goes to the appropriate model.
# Router prompt (works with cheap model) Classify the incoming request according to its difficulty. Return only the following JSON:{"difficulty": "simple|complex"}Simple: single-step, formulaic, short-answer.Complex: requiring multi-step reasoning, analysis, or long generation.Request: """{{request}}"""
- go to simple → fast model (cheap, fast).
- go to complex → powerful model (expensive but necessary).
This pattern significantly reduces the average cost because most of the traffic is generally simple.
Tip: A referral decision doesn't always require an LLM. Simple rules like “Go to fast model if text is less than 20 words” are also a guide and bring zero additional token cost. Try the rule first.
Linking Choice to Evidence: The Small Eval Cluster
Do not choose a model based on "it looks better to me". Eval (evaluation set) is a small set of samples for which the correct answer is known; you run each model on this set and measure accuracy, cost, and latency.
# Eval setup template1) Collect 20-50 real examples, hand write the "correct answer" on each.2) Run each model (fast/balanced/strong) on this set.3) For each model: number of corrects, average throughput tokens, cost per request, average time.4) Choose the model that "gives sufficient accuracy cheapest".
# Eval comparison table (fill)Model | Accuracy | Cost per request | Average durationHaiku | ...% | ... $ | ... snSonnet | ...% | ... $ | ... snOpus | ...% | ... $ | ...sec
Weak prompt / Strong prompt (model selection decision)
# WEAK (no basis for decision)Let's use the best model, budget is not important.
# STRONG (decision based on measurement) In eval of 50 samples, Haiku gave 96% accuracy, Sonnet gave 97% accuracy; The difference is statistically insignificant. Haiku was chosen because it is 5 times cheaper and 2 times faster. If the accuracy drops below 95%, the decision to upgrade to Sonnet will be made automatically.
Powerful version; binds the selection to a number, a threshold, and an escalation rule. This both defends today's decision and manages future change.
Three Mini Cases
Case 1 — Escape from the overpowering model. A call center was producing all conversation summaries with Opus; monthly bill was high. At 40-sample eval, Sonnet was 1% behind Opus in accuracy but cost one-third. They moved the summary work to the Sonnet; monthly cost dropped from $9,000 to $3,100, with no quality complaints.
Case 2 — Mixed traffic with redirection. 80% of a legal tech team's requests were simple document tagging, 20% were complex contract analysis. They were sending them all to the powerful model. They added a cheap router and distributed simple jobs to Haiku and complex jobs to Opus; average request cost dropped by 64%, while analysis quality was maintained.
Case 3 — The cost of downsizing without measuring. To reduce cost, one team reduced complex medical code extraction directly to the fast model; They did not eval. In live, accuracy dropped from 92% to 78%, resulting in a return of incorrect inferences. They had to eval first: that task required the powerful model. Lesson: both reduction and elevation are done by measurement.
Common mistakes
- The “strongest model” reflex: Waste and unnecessary delay in simple tasks.
- Changing the model without measuring: Both reduction and enlargement are risky without eval.
- Locking into a single model: Routing in mixed traffic is often more efficient.
- Always mistaking the router for LLM: Simple rules can work at zero cost.
- Not setting a boost threshold: What happens if accuracy drops should be defined in advance.
- Not fixing the model version: Record which model/version you are working on in production; Version change may shift behavior.
Deeper: Perpetuating Eval and Incremental Trial
Model selection is not a one-time decision. Providers introduce new models, prices change, your job description evolves. So set up the eval cluster once and don't forget; hold it like a living being. When a new model comes out, you run the same 20-50 samples through it, update the table, and make your decision again. This protects you from the “pattern-switching intuition” trap.
The second advanced technique is the fallback / cascade pattern. You give the task to the cheap model first; If the output has low confidence or the verification layer (unit 11) rejects it, you escalate the same request. So most of the traffic is resolved on the cheap model, with only the remaining minority going to the expensive model. This is both cheaper and more durable than the fixed single-model approach.
The third point is that eval includes not only accuracy but also cost and latency. If a model is 1% more accurate but 3 times more expensive and 2 times slower, the trade-off isn't worth it for most jobs. Make the decision along three axes (accuracy, cost, latency) and define a “sufficiency threshold”: “if accuracy is above 95%, choose the cheapest.”
Finally, record which model/version you used in production. If one day the output quality changes, the first thing you will look at is whether the model version has changed. Version traceability makes it faster to find the root cause of quality issues.
One more caveat: the eval cluster should represent your actual workload. An eval consisting only of easy examples hides where the model stumbles in difficult cases and lulls you into false confidence. A good eval; It includes common easy examples as well as corner cases you encounter in reality (ambiguous, incomplete, contradictory inputs). This difficult minority determines your model choice, because every model succeeds in the easy majority anyway. Keep your Eval fresh and representative by periodically feeding it new real examples.
In summary
The right model is the lightest model that gets the job done; More powerful isn't better at every job, it's just more expensive and slower. Classifying the task and starting from the lightest candidate, distributing the mixed traffic with routing, and substantiating the selection with a small set of evals reduces the cost many times while maintaining quality.
Application task
Choose a workload. (1) Classify the task as simple/complex. (2) Design a small eval set of 20 real examples (with their correct answers). (3) Draw up a plan to populate the accuracy/cost/time comparison table for the three model classes. (4) If you have mixed traffic, write a routing rule and set an escalation threshold.
checklist
- [ ] I can compare the model family on the capability/speed/cost axis.
- [ ] I can apply the "lightest successful model" principle.
- [ ] I can set up model routing according to task complexity.
- [ ] With a small set of eval I can bind the selection to the evidence.
- [ ] I can define an upgrade/demotion threshold.