Fine-Tuning vs RAG vs Prompting: What Hardware Each Actually Needs
Fine-tuning, RAG and prompting are three ways to make a large language model useful for your business, and each one puts a very different load on your hardware. Prompting needs only enough GPU memory to run the model. RAG adds a retrieval layer that mostly lives on CPU, RAM and fast storage. Fine-tuning is the only one that actually trains the model, and it can need several times more GPU memory than running it. Choosing the method before choosing the server is the single best way to avoid overspending. This guide explains what each approach needs, with practical configurations for teams moving AI workloads on-premise.
The three approaches in one paragraph each
Prompting means you use a pre-trained model as it is and steer it with instructions and examples inside the prompt. Nothing about the model changes. RAG (Retrieval-Augmented Generation) keeps the model unchanged but feeds it relevant passages from your own documents at query time, so answers are grounded in your data. Fine-tuning updates the model’s weights on your examples so it learns a style, format or narrow skill permanently.
Most production systems combine them: a well-prompted model, with RAG for knowledge, and fine-tuning only where behaviour must change.
Hardware for prompting: inference only
Prompting is pure inference. The main constraint is GPU memory (VRAM), which must hold the model weights plus the KV cache for every active conversation. A rough rule:
- FP16/BF16 weights need about 2 GB per billion parameters.
- 8-bit quantization halves that; 4-bit quantization cuts it to roughly 0.5–0.6 GB per billion parameters.
- Long prompts and many concurrent users grow the KV cache quickly, so leave 20–40% headroom.
An 8B model in 4-bit runs comfortably on a single 24 GB card; a 70B model in 4-bit needs around 40–48 GB before cache, which in practice means a 48 GB or 80 GB GPU, or two cards. For a deeper breakdown, see our guide to cost per token on inference GPUs.
Hardware for RAG: inference plus a retrieval stack
RAG uses the same inference GPU as prompting, but adds three components with their own needs:
- Embedding model: small (typically under 1–2 GB) and fast. It can share the inference GPU or run on a smaller card. Bulk indexing of millions of documents benefits from a dedicated GPU.
- Vector database: engines such as Qdrant, Milvus or pgvector keep indexes largely in RAM. Plan 64–256 GB of system memory depending on corpus size, and plenty of CPU cores for search and document parsing.
- Storage: NVMe for the index, model files and source documents. Ingestion pipelines (PDF parsing, OCR) are CPU-heavy, not GPU-heavy.
Retrieved passages also make prompts longer, which increases KV-cache use. A RAG deployment serving the same number of users as a plain chatbot typically needs more VRAM headroom, not a bigger model. If your documents cannot leave your network, our article on air-gapped on-prem LLMs covers the isolation side.
Hardware for fine-tuning: where memory explodes
Training stores far more than weights. Full fine-tuning keeps weights, gradients and optimizer states (Adam keeps two extra values per parameter), plus activations. In mixed precision this adds up to roughly 16–20 bytes per parameter before activations, so full fine-tuning a 7B model can exceed 100 GB of GPU memory, and a 70B model needs a multi-GPU node.
Parameter-efficient methods change the picture dramatically:
- LoRA freezes the base model and trains small adapter matrices. Memory is close to inference plus a modest overhead.
- QLoRA loads the frozen base model in 4-bit and trains LoRA adapters on top. A 7–8B model can be fine-tuned on a single 24 GB GPU, and a 70B model on one 80 GB GPU or two 48 GB cards, with smaller batch sizes.
For exact numbers on large models, read how much memory fine-tuning a 70B LLM needs.
Side-by-side comparison
| Prompting | RAG | Fine-tuning (QLoRA) | Full fine-tuning | |
|---|---|---|---|---|
| Changes the model? | No | No | Adapters only | All weights |
| Main bottleneck | VRAM for weights + cache | VRAM + system RAM + CPU | VRAM | VRAM and GPU interconnect |
| 8B model, typical setup | 1× 24 GB GPU | 1× 24–48 GB GPU, 64–128 GB RAM | 1× 24 GB GPU | 2–4× 80 GB GPUs |
| 70B model, typical setup | 1× 80 GB or 2× 48 GB (4-bit) | Same, plus 128–256 GB RAM | 1× 80 GB or 2× 48 GB | 8-GPU node with NVLink |
| Best for | General tasks, fast start | Answers from changing company knowledge | Style, format, narrow skills | Research, deep domain shift |
Which approach should you buy hardware for?
- Your problem is “the model doesn’t know our documents”: choose RAG. Fine-tuning is a poor way to teach facts that change.
- Your problem is “answers are in the wrong format or tone”: try better prompting first, then LoRA/QLoRA fine-tuning.
- You need both: size the server for RAG inference in production, and run fine-tuning jobs on the same GPUs off-hours or on a separate training box.
A practical path for most teams: start with one inference server sized for the largest model you will serve, add RAG on the same machine, and only invest in a multi-GPU training node once a fine-tuning use case has proven its value. Browse our enterprise GPU server comparison to match configurations to these tiers.
Common sizing mistakes
- Buying for full fine-tuning when QLoRA would meet the goal at a fraction of the memory.
- Sizing VRAM for the model weights only and forgetting the KV cache for long RAG prompts.
- Putting the vector database on a server with too little RAM, which forces slow disk-based search.
- Ignoring GPU interconnect: multi-GPU training benefits greatly from NVLink, while inference of a model that fits on one card does not.
Frequently asked questions
Is RAG cheaper than fine-tuning?
Usually yes. RAG needs only inference hardware plus RAM and storage for the index, and documents can be updated without retraining. Fine-tuning adds training compute and must be repeated when the knowledge changes.
Can I fine-tune a 70B model on a single GPU?
With QLoRA, a 70B model can be fine-tuned on a single 80 GB GPU using small batch sizes. Full fine-tuning of a 70B model requires a multi-GPU node.
Does RAG need a separate GPU for embeddings?
Not necessarily. Embedding models are small and can share the inference GPU. A dedicated GPU helps when you index very large document sets or re-index frequently.
Which matters more for RAG: GPU or RAM?
Both, for different parts. The GPU determines how fast answers are generated; system RAM determines how large a vector index you can search quickly. Undersizing either one creates a bottleneck.
Conclusion
Prompting, RAG and fine-tuning are not competing products but different tools, and each maps to a different hardware profile. Decide which problem you are solving, size VRAM for the largest model and context you will serve, add RAM and NVMe for retrieval, and reserve multi-GPU training hardware for proven fine-tuning needs. If you want a configuration matched to your models and user count, the ITCT team can help you compare options.
Last update at December 2025