To choose an LLM, a team should test candidate models against one real task and compare 3 numbers: the task completion rate, the runtime per task, and the total cost per completed task. The model that clears your quality bar at the lowest cost per completed task is the right choice for that task. Token price and leaderboard rank are inputs to that decision, not the decision itself.
Most selection guides rank models in general. That is useful background, but your team does not run "general" work. It runs specific tasks, such as triaging support tickets, drafting release notes, or reviewing pull requests. The rest of this guide shows how to turn a vague model choice into a decision you can defend to finance and to engineering.
Why token price is the wrong starting point
Token price tells you what one unit of input or output costs. It does not tell you what one finished piece of work costs. A cheaper model that fails 1 task in 3 forces retries, human fixes, or a fallback call to a bigger model. Each of those adds spend that never shows up in the price sheet.
Prices also move fast, which makes any single price comparison short-lived. Epoch AI found that the cost of reaching a given level of AI performance has fallen about 47% per quarter since 2023, roughly 13x per year (Epoch AI, The plunging price of thought). An earlier Epoch analysis showed the rate of decline varies widely by task, from 9x to 900x per year depending on the benchmark milestone (Epoch AI, LLM inference price trends). At the same time, researchers at MIT FutureTech estimate that the price of running frontier models is rising 3x to 18x per year, because models are bigger and reasoning uses more tokens (Gundlach et al., The Price of Progress).
The practical lesson is that "cheap" and "expensive" are moving targets. The unit that stays meaningful is the completed task. This is the core of token economics.
The 3 numbers that decide a model choice
Task completion rate
Task completion rate is the share of attempts where the output is usable without human repair. Define "usable" before you test anything. Anthropic's evaluation guidance recommends success criteria that are specific and measurable: "accurate sentiment classification" rather than "good performance" (Anthropic, Define success criteria and build evaluations).
Write this definition down. Without it, teams end up judging models by gut feel, and the most persuasive reviewer wins instead of the best model.
Runtime per task
Runtime is the time from request to usable result, including retries. Anthropic lists response time alongside accuracy as a core operational metric for LLM applications (Anthropic). For user-facing features, LLM latency shapes how the product feels. For background jobs, runtime matters less per call but adds up across large volumes.
Measure the full path, not just the first response. A model that answers in 2 seconds but needs a second attempt 30% of the time has a longer effective runtime than its first reply suggests.
Total cost per completed task
Total cost per completed task is the full spend divided by the number of usable results. Include input tokens, output tokens, retries, fallback calls, and the human time spent fixing failed outputs. This number makes an AI model cost comparison honest, because it puts cheap-but-unreliable and expensive-but-reliable models on the same scale.
A step-by-step LLM evaluation framework
Step 1: pick one task and define success
Choose a task with real volume, a clear owner, and a measurable outcome. Write a 1-paragraph definition of a usable result and agree on it with the person who owns the task. Starting with 1 task keeps the evaluation small enough to finish in days, not months.
Step 2: build a test set from real work
Collect real examples of the task with the expected output for each. Practitioner guidance commonly recommends a "golden set" of 50 to 200 real cases that becomes the regression suite for every prompt or model change (Data AI Hub, LLM Evaluation Guide). Include easy cases, typical cases, and the awkward edge cases that cause trouble in production.
Step 3: shortlist 3 to 4 candidate models
Pick candidates across price tiers: at least 1 small model, 1 mid-tier model, and 1 large model. Include an open source option if data residency, cost control, or self-hosting matters. Keep the prompt identical across models so you are testing the model, not the prompt.
Step 4: run the test set and record the 3 numbers
Run every example through every candidate. Record completion, runtime, and total cost for each run, then calculate the cost per completed task.
| Model tier | What to record | Pattern to watch for |
|---|---|---|
| Small | Completion rate, retries | Strong on narrow, structured tasks; drops on reasoning |
| Mid-tier | Completion rate, runtime | Often a good balance for drafting and extraction |
| Large | Cost per completed task | Highest quality; worth it only when quality drives value |
Step 5: choose the cheapest model that clears the bar
Discard any model below your completion threshold. Among the rest, pick the one with the lowest cost per completed task, unless runtime rules it out for a user-facing feature. Record why you chose it, so the decision can be revisited when prices or models change.
How to read public benchmarks without over-trusting them
Public LLM benchmarks are a good way to build a shortlist. They are a poor way to make the final decision. One reason is data contamination: benchmark questions leak into training data, so a model can score well by memory rather than skill.
An EMNLP 2024 study of more than 15 popular models found contamination levels from 1% to 45% across 6 widely used benchmarks, with accuracy inflated by up to 14% on one contaminated benchmark (Li et al., An Open-Source Data Contamination Report). A 2026 review of 55 studies reported inflation estimates of roughly 6% to 40% and concluded that no detection method is consistently reliable (Nourbakhsh et al., Are LLM Benchmarks Already Contaminated?).
Treat a benchmark score as a reason to include a model in your test, not as a reason to deploy it. The result that counts is how the model performs on your own examples.
When to route between models instead of choosing one
Some workloads mix easy and hard requests. In that case, routing can beat a single choice: send simple requests to a small model and escalate complex ones to a larger model. Researchers from UC Berkeley, Anyscale and Canva showed that a trained router can cut costs by more than 2x on public benchmarks without reducing response quality (Ong et al., RouteLLM).
Routing adds engineering work and needs guardrails, such as a clear escalation rule and logging of which model handled each request. Start with a single model per task, and add routing only once your cost-per-task data shows where spend concentrates.
Making the choice stick
A model decision is not permanent. Given how quickly prices fall for a fixed level of performance, the right choice this quarter may not be the right choice next quarter (Epoch AI). Re-run your test set on a fixed schedule and whenever a major new model is released. Because the test set and success definition already exist, a re-check takes hours rather than weeks.
Keep an audit trail of each evaluation: the date, the candidates, the 3 numbers for each, and the final choice. That record turns model selection from a debate into a repeatable process, and it gives finance a clear answer when they ask why a given model is in production.
Conclusion
Choosing an LLM is a task-level decision. Define a usable result, test 3 to 4 candidates on real examples, and compare task completion, runtime, and total cost per completed task. Public benchmarks help you shortlist, but contamination research shows they can overstate real performance. The right model is the cheapest one that clears your quality bar for that specific task, and that answer should be re-checked as prices keep falling.
Frequently asked questions
How should a team choose an LLM?
Define what a usable result looks like for one real task, using success criteria that are specific and measurable (Anthropic, Define success criteria and build evaluations). Then test shortlisted models on a golden set of 50 to 200 real cases (Data AI Hub, LLM Evaluation Guide) and choose the cheapest model that clears your quality bar.
What LLM evaluation metrics matter most?
Task completion rate, runtime per task including retries, and total cost per completed task. Anthropic lists response time alongside accuracy as a core operational metric for LLM applications (Anthropic, Define success criteria and build evaluations).
Are public LLM benchmarks enough to choose a model?
No. They help build a shortlist, but benchmark questions leak into training data. An EMNLP 2024 study found contamination levels from 1% to 45% across 6 widely used benchmarks, with accuracy inflated by up to 14% (Li et al., An Open-Source Data Contamination Report). Make the final choice from your own test set.
How often should a team re-evaluate its LLM choice?
On a fixed schedule and whenever a major new model is released. The cost of reaching a given level of AI performance has fallen about 47% per quarter since 2023, so the right choice this quarter may not be the right one next quarter (Epoch AI, The plunging price of thought).