Air-Gapped On-Prem LLM for Banks and Government in the Gulf

Every bank and government entity in the Gulf is being asked the same question this year: can we use large language models on our own data without that data leaving our control? The honest answer is yes, and the honest follow-up is that “on-premise” and “air-gapped” are not the same thing, and the gap between them is where most procurement goes wrong. This guide covers what an air-gapped LLM deployment actually requires, what it costs in hardware, and where the real operational burden sits.

Four deployment models, in order of isolation

Model	Where data goes	Typical fit
Public API	Vendor’s cloud, outside the region	Non-sensitive workloads only
In-region cloud	A hyperscaler region inside the country	Many commercial use cases
On-premise, connected	Your datacentre, but the stack still calls out for licensing, telemetry or model updates	Common, and often mislabelled as air-gapped
Air-gapped	Your datacentre, zero external dependencies	Classified work, core banking, defence

The third row is the trap. Several products marketed as on-premise still require connectivity for license validation or model serving. If your security posture assumes no outbound traffic, that requirement surfaces at the worst possible moment, usually during an audit.

What air-gapped actually means in procurement terms

Write these into the requirement document, because vendors interpret the word differently:

• No outbound network calls of any kind in normal operation, including telemetry, crash reporting and usage metering.
• Offline licensing. No call-home for activation or renewal, and a defined process for license files.
• Offline model updates. Weights and container images delivered as signed artefacts you transfer in, with published checksums.
• Offline dependency resolution. A local registry and package mirror, because a container that pulls from the public internet on first start is not air-gapped.
• Local inference only. No fallback to a hosted endpoint when the local model is busy or fails.
• Self-contained observability. Logging and monitoring that terminate inside the boundary.

The hardware question

Model size drives everything. Memory is the binding constraint, not raw compute, and the useful figure is the total GPU memory needed to hold the weights plus the key-value cache for your concurrency.

• Small models (roughly 7B to 13B): run comfortably on a single workstation-class or entry server GPU. Suitable for classification, extraction, routing and simple summarisation.
• Mid-size (around 30B to 70B): typically a multi-GPU server. This is where most document-heavy enterprise work lands, and where quantisation decisions start to matter.
• Large frontier-class open models: a full multi-GPU system with high-bandwidth interconnect between cards, because tensor parallelism is bandwidth-bound.

Two practical notes. First, serving concurrency costs memory beyond the weights, and teams routinely undersize for this; a model that fits in theory will not serve forty simultaneous users. Second, quantisation reduces memory substantially and costs some accuracy, so test it against your own documents rather than a public benchmark. See 70B fine-tuning memory requirements for the arithmetic, and cost per token for choosing between cards.

Retrieval matters more than the model

For banks and government, the use case is almost always answering questions over internal documents: policies, circulars, contracts, case files. That means the retrieval layer determines quality far more than which open model you picked.

• Your embedding model must also be local. It is easy to forget that calling a hosted embedding API sends the document text out. This is the single most common leak in otherwise careful deployments.
• Arabic and bilingual retrieval needs testing. Many embedding models underperform on Arabic, and Gulf institutions typically hold mixed Arabic and English corpora with inconsistent formatting. Benchmark on your own documents.
• Document access control has to be enforced at retrieval time, not in the prompt. If a user cannot open a file in the document system, the retriever must not return its contents.
• Scanned documents need OCR, and OCR quality sets a ceiling on everything downstream.

The operational burden nobody budgets for

The hardware is the visible cost. The running cost is people and process.

1. Model updates are a project each time. Validating a new model version offline, re-testing against your evaluation set, and rolling it out is weeks of work, not an afternoon.
2. Evaluation must be built in-house. You cannot rely on public leaderboards to tell you whether the system answers your circulars correctly. Build a question set with known answers and re-run it on every change.
3. Capacity planning is yours. There is no elastic scaling. Peak concurrency has to be sized for, and sized conservatively.
4. Someone must own the transfer process for bringing artefacts across the boundary, with signature verification and an audit trail.
5. Hardware lifecycle. Spares, firmware, and a plan for what happens when a GPU fails in a system with no remote vendor access.

Where to start if the mandate just landed

1. Classify the data first. Most organisations find that only a minority of use cases genuinely need full air-gapping. Putting everything behind the strictest tier makes the programme slower and more expensive than the risk justifies.
2. Pick one document-heavy use case with a measurable baseline, such as how long staff currently take to find an answer in policy documents.
3. Start with a mid-size model and good retrieval rather than the largest model you can afford. Retrieval quality moves the result more than parameter count at this stage.
4. Prove the offline path early. Do the artefact transfer, the offline licensing and an offline model update during the pilot, not after.
5. Build the evaluation set before the system, because it is the only way you will know whether changes help.

Frequently asked questions

What is the difference between on-premise and air-gapped?

On-premise means the hardware sits in your datacentre. Air-gapped means there are no external network dependencies at all. Many on-premise products still call out for licence validation, telemetry or model updates, which breaks an air-gap requirement. Ask vendors to confirm zero outbound traffic in writing.

How much GPU memory do we need for an air-gapped LLM?

Memory is the binding constraint, not compute. You need enough total GPU memory for the model weights plus the key-value cache for your expected concurrency. A 7B to 13B model runs on a single server GPU; 30B to 70B typically needs a multi-GPU server; frontier-class open models need high-bandwidth interconnect between cards. Teams most often undersize for concurrency rather than for the weights.

Does quantisation reduce accuracy too much for banking use?

It reduces memory substantially and costs some accuracy, but how much depends on your documents and task. Test it against your own evaluation set rather than a public benchmark, because results on general benchmarks do not transfer reliably to domain-specific retrieval over internal policies and circulars.

Can we use a hosted embedding model with a local LLM?

Not if you need an air-gap. Calling a hosted embedding API sends your document text outside the boundary, which defeats the purpose. The embedding model must run locally too. This is the most common leak we see in otherwise careful deployments.

Does air-gapping make the system secure?

It protects against remote compromise, not against physical or insider access. Treat it as one control among several, alongside access control enforced at retrieval time, audit logging inside the boundary, and a controlled process for transferring artefacts in.

A note on claims

Market figures circulating about on-premise AI growth come largely from vendor publications and are not independently verified, so treat them as directional. The regulatory driver is more concrete: where financial or government data is subject to residency and control requirements, keeping inference inside the boundary is a compliance question rather than a preference. The specifics differ by regulator and by data class, which is why the classification step above comes first.

One security caveat worth stating plainly: air-gapping protects against remote compromise, not against physical or insider access. It is one control among several, not a complete security posture.

For the surrounding infrastructure decisions, see GPU cluster design and enterprise GPU server comparison.

---

Last update at December 2025