How Much Memory Do You Need to Fine-Tune a 70B LLM Locally?
Fine-tuning a 70-billion-parameter model on your own hardware sounds like something only a data center can do. In 2026 that is no longer true — but the amount of memory you need changes by a factor of more than twenty depending on how you fine-tune. A full fine-tune of a 70B model needs well over a terabyte of GPU memory. A QLoRA fine-tune of the same model can fit in under 64 GB. This guide works through the numbers step by step, so you can see exactly which method fits which machine before you spend money on hardware.
The short answer
| Method (70B model) | Approximate memory needed | What it runs on |
|---|---|---|
| Full fine-tune (BF16 + AdamW) | 1.1–1.4 TB | Multi-GPU server: 8× H200 / B200 class |
| LoRA (BF16 base weights) | 160–200 GB | GB300 workstation, 2–3× H200, 2× RTX PRO 6000 (tight) |
| QLoRA (4-bit base weights) | 45–65 GB | DGX Spark (128 GB), one 80–96 GB GPU, one 48 GB GPU (very tight) |
| Inference only (4-bit) | 40–50 GB | Almost any 48 GB+ card or unified-memory machine |
The ranges are wide because sequence length, batch size and gradient checkpointing all move the number. The sections below show where each figure comes from.
Where the memory goes during fine-tuning
Training memory is made of four parts. Understanding them is the key to every number in this article.
- Model weights. 70 billion parameters × bytes per parameter. In BF16 that is 2 bytes each, so about 140 GB. Quantized to 4 bits it is about 0.5 bytes each, so about 35–40 GB including quantization constants.
- Gradients. One value for every parameter you train. If you train all 70B parameters in BF16, that is another 140 GB.
- Optimizer states. AdamW keeps two extra values per trained parameter, normally in FP32 (4 bytes each), plus often an FP32 master copy of the weights. That adds roughly 12 bytes per trained parameter — about 840 GB for a 70B model.
- Activations. The intermediate results saved during the forward pass so the backward pass can use them. These grow with batch size × sequence length × model width, and can range from a few gigabytes to hundreds. Gradient checkpointing cuts them sharply at the cost of roughly 20–30% more compute.
A useful rule of thumb: full fine-tuning with AdamW in mixed precision needs about 16–20 bytes per parameter before activations. For 70B parameters, that is 1.1–1.4 TB.
Full fine-tuning a 70B model: a cluster job
Because every parameter is trained, full fine-tuning carries the full weight of gradients and optimizer states. Even with ZeRO-3 or FSDP sharding, which spreads those states across GPUs, the total still has to live somewhere.
- 8× H100 80 GB (640 GB total) cannot hold a 70B full fine-tune without CPU or NVMe offload, which makes training very slow.
- 8× H200 141 GB (1,128 GB total) is the practical starting point, with sharding and gradient checkpointing.
- 8× B200 / B300 gives more headroom and much higher throughput.
Full fine-tuning is worth it when you are changing the model’s behaviour deeply — new languages, a new domain vocabulary, or continued pre-training on large private datasets. For most business use cases, parameter-efficient methods reach similar quality at a fraction of the memory. For hardware at this level, see our HGX platform guide.
LoRA: train 1% of the parameters
LoRA (Low-Rank Adaptation) freezes the original weights and trains small adapter matrices added to the attention and MLP layers. With a typical rank of 16–64, the adapters hold roughly 0.1–1% of the model’s parameters — a few hundred million for a 70B model.
That changes the maths completely:
- Frozen base weights in BF16: ~140 GB
- Adapter weights, gradients and AdamW states: ~3–10 GB
- Activations with gradient checkpointing, 4K context: ~15–40 GB
Total: roughly 160–200 GB. That rules out any single 80 GB or 96 GB card, but fits comfortably in one machine with a large memory pool — and leaves room for longer sequences.
A GB300 workstation with 748 GB of coherent memory (252 GB of it HBM3e on the GPU) can run a 70B LoRA in BF16 with plenty of headroom, and can even hold a second model for evaluation at the same time. Two H200 cards (282 GB) also work well. Two 96 GB RTX PRO 6000 cards (192 GB) can do it, but only with short sequences and careful tuning.
QLoRA: 70B on a single desktop
QLoRA goes one step further: it stores the frozen base model in 4-bit NF4 format and dequantizes layers on the fly while training BF16 LoRA adapters on top. Quality loss compared with standard LoRA is small in most published tests, and the memory saving is dramatic.
- 4-bit base weights: ~35–40 GB
- LoRA adapters, gradients, optimizer (paged): ~3–8 GB
- Activations with checkpointing: ~5–20 GB
Total: about 45–65 GB. This is the method that brings 70B fine-tuning to a desk.
- A 48 GB card (RTX 6000 Ada, L40S, A6000) can just fit a 70B QLoRA with short sequences and batch size 1. It works, but leaves almost no headroom.
- An 80–96 GB card (H100, RTX PRO 6000 Blackwell) runs it comfortably.
- An NVIDIA DGX Spark with 128 GB of unified memory fits a 70B QLoRA with room for longer context. NVIDIA itself lists fine-tuning of models up to 70B as a target workload for Spark.
Memory is not the same as speed
Fitting a job in memory only tells you it will run — not how long it will take. Training speed depends mostly on memory bandwidth and compute.
| Machine | Memory | Memory bandwidth | 70B QLoRA | 70B LoRA (BF16) |
|---|---|---|---|---|
| DGX Spark (GB10) | 128 GB unified | ≈ 273 GB/s | Fits, slow — good for small datasets and experiments | Does not fit |
| 1× RTX PRO 6000 Blackwell | 96 GB GDDR7 | ≈ 1.8 TB/s | Fits, fast | Does not fit |
| 1× H200 NVL | 141 GB HBM3e | ≈ 4.8 TB/s | Fits, very fast | Does not fit alone |
| GB300 workstation | 748 GB coherent | ≈ 7 TB/s (HBM3e) | Fits, very fast | Fits with headroom |
| 8× H200 server | 1,128 GB HBM3e | ≈ 4.8 TB/s per GPU | Overkill | Fits; full fine-tune possible |
In practice, DGX Spark is the lowest-cost way to prove that a 70B fine-tune works on your data — to test prompts, data formats and hyperparameters on a small sample. When the dataset grows to hundreds of thousands of examples, a machine with HBM bandwidth finishes the same job many times faster. Our DGX Spark comparison covers Spark’s strengths and limits in more detail.
Worked example: a 70B legal assistant
Say a law firm wants to fine-tune Llama 3.3 70B on 20,000 internal question-and-answer pairs averaging 2,000 tokens each, and the data must not leave the building.
- Method: QLoRA, rank 32, sequence length 4,096, gradient checkpointing on.
- Memory: about 50–60 GB.
- On DGX Spark: fits. One epoch over 40 million tokens takes on the order of days — fine for a first version, slow for weekly retraining.
- On one H200 or RTX PRO 6000: the same epoch takes roughly hours, not days.
- On a GB300 workstation: fast enough to switch to full-precision LoRA, run several experiments a day and serve the result to the team from the same machine.
The pattern is common: start small to validate, then move to bandwidth when the project proves its value.
Six ways to cut fine-tuning memory
- Gradient checkpointing. Recomputes activations instead of storing them. Often halves activation memory for a 20–30% slowdown.
- Shorter sequences. Activation memory grows with sequence length; packing short examples together keeps GPUs busy without long padding.
- Lower LoRA rank. Rank 8–16 is often enough for style and format tuning; rank 64+ helps for new knowledge.
- 8-bit or paged optimizers. 8-bit AdamW cuts optimizer memory by about 75%; paged optimizers spill to CPU memory during spikes instead of crashing.
- FlashAttention. Reduces attention memory from quadratic to linear in sequence length and speeds training.
- Sharding across GPUs. FSDP or DeepSpeed ZeRO splits weights, gradients and optimizer states over several cards — essential for full fine-tuning.
Which machine should you buy?
- Experimenting, one developer, budget-sensitive: DGX Spark. Fits 70B QLoRA and 200B-class inference, sits on a desk, uses the same NVIDIA software stack as larger systems.
- A team fine-tuning regularly: a workstation with one or two 96 GB RTX PRO 6000 cards or H200 NVL cards — see our GPU comparison.
- Serious 70B LoRA, several users, sensitive data on-premise: a GB300 workstation. One coherent memory pool, HBM3e bandwidth, no rack required. Our GB300 buyer’s guide compares the main models.
- Full fine-tuning or continued pre-training: an 8-GPU H200 or B200 server — see how to buy an AI server.
Not sure which fits your workload? Send us your model size, dataset size and how often you retrain on WhatsApp, and we will size it with you.
Frequently asked questions
How much VRAM do I need to fine-tune a 70B model?
About 45–65 GB with QLoRA, 160–200 GB with standard LoRA in BF16, and 1.1–1.4 TB for a full fine-tune with AdamW. The exact figure depends on sequence length, batch size and whether gradient checkpointing is enabled.
Can I fine-tune a 70B model on a single GPU?
Yes, with QLoRA. A single 80–96 GB GPU handles it comfortably, and a 48 GB GPU can just fit it with short sequences and batch size 1. Standard LoRA and full fine-tuning need more memory than any single PCIe card offers.
Can DGX Spark fine-tune a 70B model?
Yes. With 128 GB of unified memory, DGX Spark fits a 70B QLoRA fine-tune. It is slower than HBM-based GPUs because of its lower memory bandwidth, so it suits experiments and smaller datasets best.
Is QLoRA as good as full fine-tuning?
For most instruction-tuning and domain-adaptation tasks, QLoRA reaches quality close to full fine-tuning. Full fine-tuning still has the edge when you need to teach the model large amounts of new knowledge or a new language.
How much memory does a 70B model need just for inference?
About 140 GB in BF16, about 70–75 GB in 8-bit, and about 40–50 GB in 4-bit, plus memory for the KV cache, which grows with context length and the number of simultaneous users.
What is the cheapest way to fine-tune a 70B model on-premise?
QLoRA on a single machine with 64 GB or more of GPU or unified memory. DGX Spark is the lowest-cost NVIDIA system that fits it with comfortable headroom.
Last update at December 2025
5 thoughts on “How Much Memory Do You Need to Fine-Tune a 70B LLM Locally?”