How Much GPU Memory Do LLMs Need? Training vs Inference Explained
The first question in almost every AI hardware conversation is the same: how much GPU memory do I actually need? Buy too little and the model will not load, or it will run with a context window so short that it is useless. Buy too much and you have tied up budget in VRAM that sits idle. The answer depends on three things: the size of the model, the numeric precision you run it at, and whether you are training, fine-tuning or only serving it. This guide gives you the formulas, worked examples for popular model sizes, and the GPU configurations that match each case.
The short answer
| Model size | Inference, FP16/BF16 | Inference, 8-bit | Inference, 4-bit | Full fine-tuning (mixed precision, Adam) |
|---|---|---|---|---|
| 7–8B | ~16 GB + KV cache | ~8 GB | ~5 GB | ~130 GB+ |
| 13–14B | ~28 GB | ~14 GB | ~8 GB | ~230 GB+ |
| 32–34B | ~68 GB | ~34 GB | ~20 GB | ~550 GB+ |
| 70B | ~140 GB | ~70 GB | ~40 GB | ~1.1 TB+ |
| 405B | ~810 GB | ~405 GB | ~230 GB | multi-node cluster |
These numbers cover the weights only (plus training state for fine-tuning). For real serving you must add memory for the KV cache, which grows with context length and the number of concurrent users. That is where most sizing mistakes happen.
Inference memory: weights plus KV cache
1. Model weights
Weights memory ≈ number of parameters × bytes per parameter. FP16 and BF16 use 2 bytes, FP8 and INT8 use 1 byte, and 4-bit formats such as AWQ, GPTQ or GGUF Q4 use roughly 0.5–0.6 bytes. A 70B model therefore needs about 140 GB in BF16, 70 GB in FP8 and around 40 GB in 4-bit.
2. KV cache
During generation, the model stores keys and values for every token in the context. A useful approximation is:
KV cache per token ≈ 2 × layers × KV heads × head dimension × bytes per value
For Llama 3 70B (80 layers, 8 KV heads with grouped-query attention, head dimension 128, FP16), that is about 320 KB per token. One user with a 32K-token context needs roughly 10 GB; 20 concurrent users at 8K tokens each need about 50 GB. Models without grouped-query attention use several times more.
3. Overhead
Add 10–20% for activations, CUDA context and framework buffers. Serving engines such as vLLM pre-allocate most of the free GPU memory for the KV cache, which is exactly what you want for throughput; we compare them in vLLM vs Ollama vs llama.cpp.
Training and fine-tuning memory
Training needs far more memory than inference, because the GPU must hold gradients and optimizer state as well as weights.
- Full fine-tuning with Adam in mixed precision: roughly 16 bytes per parameter (BF16 weights and gradients, FP32 master weights and two Adam moments), before activations. A 7B model needs about 112 GB plus activations, so it will not fit on a single 80 GB GPU without sharding.
- LoRA: the base model is frozen and only small adapter matrices are trained. Memory is close to inference memory plus a few GB, so a 7–8B model fine-tunes comfortably on a 24–48 GB GPU.
- QLoRA: the frozen base model is loaded in 4-bit. A 70B model can be fine-tuned on a single 80–96 GB GPU, at the cost of slower training.
- Activations: grow with batch size and sequence length. Gradient checkpointing trades extra compute for much lower activation memory.
- Sharding: DeepSpeed ZeRO and PyTorch FSDP split weights, gradients and optimizer state across GPUs, which is how full fine-tuning of 70B models is done on 8-GPU servers.
Worked examples
| Use case | Memory estimate | Suitable hardware |
|---|---|---|
| Internal chatbot on an 8B model, 10 users, 8K context, FP16 | 16 GB weights + ~10 GB KV + overhead ≈ 30 GB | Single 48 GB GPU such as L40S or RTX 6000 Ada |
| 70B model in FP8 for a team of 20–30, 8K context | 70 GB weights + ~60 GB KV ≈ 140 GB | 2× H100 80 GB, 1× H200 141 GB (tight), or 2× RTX PRO 6000 Blackwell 96 GB |
| 70B in BF16 for production serving with long context | 140 GB weights + 100 GB+ KV | 4× H100/H200 with NVLink |
| QLoRA fine-tuning of a 70B model | ~45 GB weights + adapters + activations | 1× H100 80 GB or RTX PRO 6000 96 GB |
| Full fine-tuning of a 70B model | >1 TB across GPUs | 8× H100/H200 HGX server with ZeRO-3 or FSDP |
Matching memory to GPUs
- 24–32 GB (RTX 4090, RTX 5090): development, 4-bit quantised models up to ~30B, LoRA on 7–8B models.
- 48 GB (L40S, RTX 6000 Ada): production inference for 7–14B models and 4-bit 70B with short context.
- 80–96 GB (H100, RTX PRO 6000 Blackwell): FP8 70B on two cards, QLoRA on 70B, high-concurrency serving of mid-size models. The RTX PRO 6000 Blackwell Server Edition is a strong value option where NVLink is not required.
- 141 GB and above (H200, B200): large models on fewer GPUs, long contexts and high batch sizes. Bandwidth matters as much as capacity for token speed.
- 8-GPU HGX and DGX systems: training and full fine-tuning of large models, see NVIDIA DGX H200 and our 8-GPU H100 server cost breakdown.
For a ranked comparison of cards by throughput and cost per token, read best GPU for LLM inference, and for current pricing see H100, H200 and B200 prices.
Common sizing mistakes
- Sizing for weights only. A model that “fits” with no room for KV cache will serve one short request at a time.
- Ignoring concurrency. KV cache scales linearly with simultaneous users and context length.
- Splitting across GPUs without NVLink. Tensor parallelism over PCIe works but loses throughput; for multi-GPU serving of a single large model, NVLink or SXM platforms help.
- Forgetting system RAM and storage. Plan system memory of at least 1.5–2× total GPU memory and fast NVMe to load checkpoints quickly.
Get a configuration sized for your model
Tell us the models you plan to run, expected users and context length, and the ITCT team will size a GPU server for you and quote available stock from Dubai. Start with our guide on how to buy an AI server or contact our engineers directly.
Frequently asked questions
How much GPU memory do I need to run a 70B model?
About 140 GB in FP16/BF16, 70 GB in FP8 and roughly 40 GB in 4-bit, plus memory for the KV cache. For a team of users, plan on two 80–96 GB GPUs in FP8 or four GPUs in BF16.
Can I run a 7B or 8B model on a 24 GB GPU?
Yes. In FP16 the weights need about 16 GB, leaving room for a moderate KV cache. With 8-bit or 4-bit quantisation you can serve more users or longer contexts.
Why does training need so much more memory than inference?
Training stores gradients and optimizer state in addition to weights, roughly 16 bytes per parameter for full fine-tuning with Adam, versus 2 bytes per parameter for FP16 inference.
Does quantisation reduce quality?
8-bit and FP8 usually have a negligible quality impact. Well-made 4-bit quantisations are close for most chat and retrieval tasks but can lose accuracy on reasoning, code and maths; test on your own workload.
Is more VRAM or more bandwidth more important?
Capacity decides what fits; bandwidth decides how fast tokens are generated. For serving large models, you need enough of both, which is why the H200 and B200 improve on the H100 in both areas.
Last update at December 2025