How to choose an AI model for a real product

Model selection is usually approached as a ranking problem: find the best model, use it. That framing produces two predictable outcomes — an expensive system doing simple work, or a cheap system that fails in ways nobody tested for.

A better framing is procurement. Constraints first, shortlist second, test third. The constraints do most of the work, and they are properties of your product rather than opinions about models.

The six properties that decide fit

Quality on your task. Not general quality. A model that writes beautiful prose may extract structured fields poorly, and a model that reasons well through mathematics may be needlessly slow at classification. General rankings are a screen for which models to test, not a substitute for testing.

Cost per request. Priced in input and output tokens, multiplied by your actual traffic. The number that matters is monthly spend at projected volume, and it is worth computing before rather than after. Note that reasoning-style models bill their internal deliberation as output tokens, so their effective cost per completed task can be several times the headline rate.

Latency. Two numbers, not one. Time to first token determines perceived responsiveness in any streaming interface. Total generation time determines throughput for batch work. An interactive product usually cares more about the first; a nightly pipeline cares only about the second.

Context window. Measure your real input distribution at the 95th percentile. Selecting for a rare maximum means paying for capacity you use a few times a month on every request you make.

Integration capabilities. Tool calling reliability, schema-enforced output, streaming behaviour, image or audio input if you need it. These vary between models far more than general quality scores imply, and they are frequently the property that decides whether an integration works at all rather than how well it works.

Deployment and data constraints. Where the data may be processed, what the retention terms are, whether a regional endpoint exists, whether the model can run in your own environment. These are often decided outside the engineering team, and they eliminate candidates regardless of every other property.

Test on your own cases

With a shortlist of two or three, the decision is settled by testing, and the test does not need to be elaborate.

Take fifty real cases from your own traffic, run each candidate, and compare per case rather than on average. The average tells you almost nothing; the interesting information is which specific cases each model gets wrong, and whether those cases matter. A model that is slightly weaker overall but never fails on the expensive category is the better choice, and no aggregate score will tell you that.

Two practical warnings. Use the same prompt for each candidate on the first pass, to get a fair baseline — then, before deciding, spend a little time tuning the prompt for the leading candidate, because models differ in what they respond to and a prompt tuned for one can understate another. And measure latency and cost during the same run, so that all three dimensions come from one experiment rather than three.

Most traffic does not need the strongest model

This is the observation with the largest financial consequence, and it is available to almost every product.

Real traffic is not uniformly difficult. A support assistant answers a large number of routine questions and a small number of genuinely hard ones. A classification pipeline sees mostly clear-cut cases. Sending all of it to the strongest available model means paying the hardest-case price for the easiest-case work.

The standard remedy is routing: a small, fast model handles the bulk, with escalation to a larger model when a confidence signal, a category, or a validation failure indicates difficulty. The saving is often large — the high-volume portion is where the money is — and the quality ceiling is preserved for the cases that need it.

Routing has a cost of its own: two models to evaluate, a routing rule to maintain, and a new failure mode where the router sends a hard case to the small model. It is worth it at volume and not worth it below some threshold. The threshold is specific to your traffic, and computing it is a half-day of work.

Choosing for change

Whatever you pick will be superseded. The models available in a year will be cheaper, faster, or better, and the decision you make now should not be expensive to revisit.

Three habits keep that cost low. Keep prompts, tool definitions and output parsing in your own code, so that switching provider is a configuration change rather than a rewrite. Keep the eval set, because it is the thing that makes evaluating a new candidate a one-day task instead of a project. And pin the model version explicitly rather than tracking a moving alias, so that a change in behaviour is something you schedule rather than something you discover.

That last one deserves emphasis. A system that follows a provider's latest alias inherits every behaviour change automatically, including the ones that break a carefully tuned prompt. Pin the version, run the eval when a new one appears, and upgrade deliberately.

The short method

1. Write down the five constraints as numbers.
2. Screen for candidates that satisfy them — usually two or three.
3. Run fifty real cases through each, measuring quality per case, latency and cost together.
4. Tune the prompt for the leader and re-run.
5. Choose the cheapest candidate that passes, not the highest scorer.
6. Pin the version, keep the eval set, and revisit when something new ships.

Step five is where most of the money is saved, and it is the step people skip, because choosing the strongest model feels safer than choosing the sufficient one. On a per-request basis, over a year, it usually is not.