Cost per Token Economics: The Real Metric for Choosing an Inference GPU
Ask most teams which GPU to buy for inference and they answer with a spec sheet: VRAM, TFLOPS, memory bandwidth. But the number your CFO actually cares about is cost per token — how many dollars it takes to generate a million tokens of output on your own traffic. That single metric quietly flips a lot of “obvious” GPU choices, and it is the reason an H100 is sometimes the wrong answer even when it is the fastest card in the rack. This guide shows how to calculate it, what moves it, and how to pick hardware around it.
What “cost per token” actually means
Cost per token is the fully loaded price of producing model output, normalized to a million tokens so you can compare options. The common working formula is simple:
Cost per 1M tokens = (hourly cost of the GPU ÷ tokens generated per second) × (1,000,000 ÷ 3,600)
The hourly cost can be a cloud rental rate or, for owned hardware, the amortized cost of the server plus power and cooling spread over its useful life. Tokens per second is your measured throughput — at your model, your batch size, and your precision — not a vendor headline number. That distinction is where most estimates go wrong.
Why the cheapest card per hour often wins
Independent 2026 benchmarks keep landing on the same uncomfortable result for smaller and mid-sized models: a cheaper card like the L40S frequently beats an H100 on cost per token, simply because its hourly price is a fraction of the H100’s while its throughput on an 8B–13B model is good enough. One analysis of Llama 3.1 8B in FP8 put the L40S well under the H100 at every batch size it tested, with the gap narrowing as batches grew. The lesson is not “L40S always wins” — it is that raw speed does not pay the bill; speed relative to price does.
The picture flips for large models. Once you are serving a 70B-class model, long contexts, or very high concurrency, the H100’s memory bandwidth and capacity let it keep GPUs busy where a cheaper card stalls or needs more units. There the H100 — or an H200/B200 — earns its premium. We break down the sticker side of that decision in our H100, H200 and B200 price guide.
Utilization is the hidden multiplier
The fastest way to destroy your cost per token is to run a powerful GPU at low utilization. At batch size 1 with naive decoding, a card can sit below 10% utilization — you are paying for the whole GPU to do a sliver of work, and system-level cost per million tokens can balloon into the tens of dollars. The fixes are mostly software: continuous batching, higher concurrency, quantization (FP8/INT8), and an inference server tuned for your traffic shape. A mid-tier GPU at 70% utilization routinely beats a flagship at 15%.
This is why “which GPU is cheapest per token” has no universal answer. It depends on model size, batch size, context length, precision, and how well you keep the card fed.
A worked example
Suppose an L40S costs about $0.90/hour to run and your tuned stack produces 900 output tokens/second on an 8B model. Cost per million tokens is (0.90 ÷ 900) × (1,000,000 ÷ 3,600) ≈ $0.28. Now take an H100 at $2.25/hour that hits 1,900 tokens/second on the same model: (2.25 ÷ 1900) × 277.8 ≈ $0.33. The H100 is faster, yet slightly more expensive per token here — so if latency is acceptable, the L40S is the better buy for this workload. Change the model to 70B and the H100’s numbers improve sharply while the L40S may not fit the model well at all. Run the same arithmetic with your own throughput before you commit.
Owned hardware changes the math again
Rental rates make cost per token easy to reason about, but if your utilization is steady and high, owning the server usually wins over a three-year horizon. The amortized hourly cost of a purchased GPU server — once you include power and cooling — is often well below cloud pricing at sustained load, which pulls your cost per token down further. We walk through that CapEx-vs-cloud comparison in the cost of an 8-GPU H100 server, and the broader buying checklist in how to buy an AI server.
How to choose, in order
- Start from the model and traffic, not the card. Model size, context length, and concurrency set the hardware envelope.
- Benchmark two or three candidates on your own workload. Measure real tokens/second at your target batch size and precision.
- Divide by price, not by spec. Convert each to cost per million tokens with the formula above.
- Check utilization before buying more power. Often the cheaper fix is batching and quantization, not a bigger GPU.
- Model owned vs rented at your real duty cycle. High, steady load favors buying; bursty or experimental load favors renting.
Get those five right and the GPU almost picks itself — on the one number that shows up in your budget.
Frequently asked questions
How do I calculate cost per token?
Divide the GPU’s hourly cost by its measured output tokens per second, then multiply by 1,000,000/3,600. Use your own benchmarked throughput, not vendor headline numbers.
Is the L40S really cheaper per token than the H100?
Often yes for small and mid-sized models (roughly 8B–13B), because its hourly cost is much lower. For 70B-class models, long context, or very high concurrency, the H100 usually wins on cost per token.
Why does utilization matter so much?
You pay for the whole GPU whether it is busy or idle. At batch size 1 with low utilization, cost per token can be tens of dollars; continuous batching, higher concurrency and quantization bring it down dramatically.
Should I rent or buy to minimize cost per token?
If utilization is steady and high, owning hardware usually gives a lower cost per token over three years once power and cooling are included. Bursty or experimental workloads favor renting.
Last update at December 2025