Sizing and Securing Offline-First AI Workstations for Law Firms: GPU Memory Maths, Isolation Options and a Staging Checklist

An independent technical review of three standard, offline-first AI workstations for law firms: the GPU-memory arithmetic behind each tier, where a real first quote was over- and under-specified, how to encrypt the machine and still restart it unattended after a power cut, how to patch and update models with no internet connection, how to keep it safe from an infected office PC, and the tests and evidence to hand over. The commercial story is in the companion blog post.

Interactive version: try the sizing calculator and see the GPU memory split for your model, context length and concurrency.

The brief

A legal-technology provider asked for three standard x86 workstations on Ubuntu 24.04 LTS Server, each running their private-AI application for law firms: document analysis and retrieval, dictation and transcription, internal knowledge search and drafting. Requirements common to all three: TPM 2.0, full-disk encryption, no permanent internet dependency, outbound traffic blocked by default, no public-facing ports, and a documented staging process that proves the application works with the network disconnected.

Tier	CPU	GPU	RAM	Storage	Users / concurrent / model
Starter	8-core (Ryzen 7 9700X class)	RTX PRO 4000 Blackwell 24 GB	32 GB	1× 4 TB NVMe	~10 / 2–3 / 8–14B
Workforce	16-core (Ryzen 9 9950X class)	RTX PRO 5000 Blackwell 48 GB	64 GB	2× 4 TB NVMe mirror	~25 / 4–6 / up to 32B
Enterprise	16–24-core Threadripper PRO / Xeon W	RTX PRO 6000 Blackwell 96 GB (Max-Q considered)	128 GB ECC	2× 4 TB NVMe mirror	25–50 / 8–12 / 70B

Sizing GPU memory from first principles

An inference server's GPU memory holds three things: the model weights, the KV cache for every request in flight, and fixed overhead (CUDA context, activations, the runtime's own buffers, plus any second model such as speech-to-text or an embedding model for search).

Weights. Parameters × bytes per parameter, plus a few percent for embeddings and quantisation scales. Roughly 0.55 bytes per parameter at 4-bit (AWQ, GPTQ, or NVFP4, which Blackwell runs natively) and 1.05 bytes at 8-bit (FP8 or INT8).

KV cache. Per token: 2 × layers × KV heads × head dimension × bytes. Modern models use grouped-query attention with 8 KV heads, which keeps this small:

Model class (example)	Layers	KV per token (FP16)	Per 8k-token request	Weights 4-bit / 8-bit
8B (Llama 3.1 8B)	32	128 KiB	1.0 GiB	4.5 / 8.5 GB
14B (Qwen2.5 14B)	48	192 KiB	1.5 GiB	8.5 / 15 GB
32B (Qwen2.5 32B)	64	256 KiB	2.0 GiB	18.5 / 34 GB
70B (Llama 3.3 70B)	80	320 KiB	2.5 GiB	40 / 74 GB

Legal work pushes context length up: a lease, a witness statement or a disclosure batch easily fills 16–32k tokens. KV cache scales linearly with context, so a 32k request costs four times the 8k figure. Paged attention (vLLM, TensorRT-LLM, SGLang) only allocates what each request actually uses, and an FP8 KV cache halves it at negligible quality cost on recent models.

Checking each tier

Tier	Workload	Weights	KV cache	Extras	Total	Fits?
Starter 24 GB	14B 8-bit, 3 × 8k	15	4.5	2 (runtime)	21.5 GB	Yes, just
Starter 24 GB	+ Whisper large-v3 transcription	15	4.5	5	24.5 GB	No: use 4-bit or 32 GB
Workforce 48 GB	32B 4-bit, 6 × 8k, + speech + embeddings	18.5	12	6	36.5 GB	Yes, with room for 16k
Workforce 48 GB	32B 8-bit, 6 × 8k	34	12	6	52 GB	No: FP8 KV gets to 46, too tight
Enterprise 96 GB	70B 4-bit, 12 × 8k, + speech + embeddings	40	30	6	76 GB	Yes
Enterprise 96 GB	70B 8-bit, 12 × 8k, FP8 KV	74	15	6	95 GB	No margin

Conclusions: the brief's GPUs are right for its stated models at 4-bit, which on Blackwell (NVFP4) costs little accuracy. The Starter is tight once transcription runs alongside chat; the 32 GB RTX PRO 4500 Blackwell is the low-cost fix. Throughput, not memory, is the open question at 8–12 concurrent requests on one GPU, and only a benchmark with the provider's own prompts and document lengths answers it. We would not offer a capacity guarantee before running one.

Reviewing the first quote

The first priced build (now listed in our shop under WSL- SKUs) went well beyond the brief:

Quoted	Versus brief	Assessment
VSX WE45-1T08: EPYC 4565P 16-core, 256 GB non-ECC, RTX PRO 5000 72 GB, 1 TB + 3.84 TB NVMe, £19,334.30	Starter brief: 8-core, 24 GB GPU, 32 GB RAM	Over-specified GPU and RAM for 2–3 users; non-ECC on an ECC-capable platform is the wrong economy
VSX WE95-1T08: EPYC 9455P 48-core, 512 GB ECC, 2× RTX PRO 5000 72 GB, £59,574.35	Workforce brief: 16-core, 48 GB GPU, 64 GB	Enterprise-plus. Two GPUs give 144 GB but a single model split across two cards over PCIe is slower than one 96 GB card
AS-5126GS-TNRT2: 2× EPYC 9555, 1.5 TB, 4× RTX PRO 6000 Server Edition, £177,485.74	No tier asked for a rack server	Right for a firm-wide or multi-office deployment, not an office appliance: 5U, loud, 2,700 W redundant supplies

Two catches that matter more than the money: both tower quotes listed 120 V APC UPS units (a UK office needs the 230 V models, for example the SMC1500IC or SMT2200IC), and the Starter used non-ECC UDIMMs on an EPYC 4005 platform that supports ECC.

CPU, memory, storage and power

• CPU. Inference is GPU-bound. The CPU handles the web application, document parsing (PDF and OCR), tokenisation and embedding queues. 8 cores is enough for the Starter, 16 for the others. The 48- and 64-core CPUs in the quote add cost and heat, and nothing else.
• System RAM. Keep at least 1–1.5× the GPU memory: models are read into page cache before loading, the vector database and document parser live in RAM, and the OS needs headroom. That makes 32 GB too small for the Starter (64 GB), 64 GB tight for Workforce (128 GB), and 128 GB fine but 256 GB comfortable for Enterprise. Use ECC everywhere: EPYC 4005, Ryzen 9000 on workstation boards, Threadripper PRO and Xeon W all support it.
• Storage. Use a mirror on every tier: mdadm RAID 1 under LUKS, or a ZFS mirror with native encryption, which also gives cheap read-only snapshots. Mirror the EFI system partition too, by keeping a second ESP synced, or the machine will not boot if the first drive dies. Use enterprise or high-endurance NVMe for the index drives; vector databases rewrite a lot.
• GPU form factor. The RTX PRO 6000 Workstation Edition draws 600 W through a flow-through cooler. The Max-Q edition is a 300 W blower card with the same 96 GB. In an office, the Max-Q's lower heat and noise outweigh its somewhat lower peak throughput, and it fits a quieter chassis and a smaller UPS.
• Power. Size the UPS from measured draw: roughly 350 W (Starter), 600 W (Workforce) and 800 W (Enterprise with Max-Q) under sustained load. A 2,200 VA pure sine wave unit gives the Enterprise well over five minutes. Use NUT (Network UPS Tools) over USB for orderly shutdown, and set the firmware to "power on after AC loss".
• Serviceability. Standardise on one chassis family, one PSU line and one NVMe model across tiers, keep spares of the drive and fans, and choose boards with a documented Linux record. A BMC (IPMI) is useful on the Enterprise tier but must sit on its own management network, never on the staff segment.

Online activation and phone-home: what actually needs the internet

Component	Needs internet?	What to do
NVIDIA RTX PRO drivers, CUDA	No activation	Install Ubuntu's precompiled, Canonical-signed modules (linux-modules-nvidia-* via ubuntu-drivers) so Secure Boot works without enrolling a MOK key
NVIDIA vGPU / AI Enterprise	Licence server	Not needed for bare-metal RTX PRO. If you buy AI Enterprise, its Delegated License Service runs on-premises
Ubuntu 24.04 LTS	No	Ubuntu Pro is optional; offline attach is possible. Disable unattended-upgrades, motd-news and apport uploads; hold or remove snaps so snapd stops trying to refresh
Docker Engine, NVIDIA Container Toolkit	No	Pre-load images with docker save/load or a local registry; pin by digest
vLLM	Usage statistics on by default	Set VLLM_NO_USAGE_STATS=1 and DO_NOT_TRACK=1
Hugging Face libraries	Try to download on start	HF_HUB_OFFLINE=1, TRANSFORMERS_OFFLINE=1; ship models in the image volume
Open WebUI (if used)	Optional features call out	OFFLINE_MODE=true; disable community sharing and web search
Model licences	No activation	Apache-2.0 (Qwen, Mistral) or community licences (Llama, Gemma): record the licence and acceptable-use terms per model in the handover pack
Firmware (UEFI, NVMe, BMC)	No	Apply with fwupdmgr install from verified .cab files during maintenance

Whatever the list says, the acceptance test is empirical: run the appliance for 24 hours with an nftables rule that logs every outbound packet and drops it, and ship only when the log is empty.

Isolating the appliance from an infected office

The realistic threat is not someone breaking in from the internet; it is a fee earner's laptop that is already compromised. Design for that:

• Own segment. The appliance sits on its own VLAN. The office firewall allows staff devices to reach exactly one port, HTTPS 443, and nothing else. Nothing on the appliance segment may start a connection back to the office.
• Default-deny on the host as well. nftables on the appliance: inbound 443 from the staff range and SSH from one admin address only; outbound dropped (option A allows only NTP to a local server).
• No file shares. Ransomware spreads over SMB. Documents enter through the application's upload or a pull from a read-only export of the document management system, never through a writable share.
• Separate identities. No Active Directory join. Users sign in to the application (OIDC from the firm's identity provider in option A, local accounts with TOTP in B and C). Administrators use SSH keys held on FIDO2 hardware keys from a dedicated admin device.
• Matter-level access. The retrieval index must respect information barriers: index per matter or per client group, enforce the firm's permissions at query time, and log every query with user and matter for audit.
• Safe file handling. Parse documents in an unprivileged, network-less container. Accept model files only as safetensors or GGUF, never pickle-based .bin or .pt files, which can run code when loaded.

Secure boot, encryption and unattended restart

The machine must come back by itself after a power cut, yet a stolen machine must be useless. The design:

1. UEFI Secure Boot on, firmware password set, boot order locked to the internal drive.
2. LUKS2 full-disk encryption on the data and OS volumes.
3. Bind the LUKS key to the TPM with Clevis (clevis-luks, clevis-tpm2, clevis-initramfs), sealed to PCR 7 (Secure Boot policy) and optionally PCR 0/2 (firmware). The disk unlocks only on that board, with that firmware and Secure Boot state.
4. For option B, add a Tang server on the internal network and require both TPM and Tang (Clevis sss with threshold 2): a machine removed from the office will not unlock even with its TPM.
5. Keep a LUKS recovery passphrase, generated at staging, printed and sealed, and delivered separately from the machine. Record the key slot layout in the handover pack.

Ubuntu 24.04's installer also offers TPM-backed encryption, but it is still marked experimental; Clevis is the well-trodden path on a server install.

Patching and model updates without an internet connection

• Packages. A staging host with internet builds a snapshot of exactly the packages the appliance uses (aptly or apt-mirror). It travels on encrypted media and is served as a local file:// repository. apt still verifies Canonical's signatures on the Release files, so a tampered package fails to install.
• Containers. Images are pulled on the staging host, scanned, and exported with their digests recorded. The appliance loads only digests on the approved list.
• Models. Download once, record the SHA-256 of every file, compare with the publisher's repository hashes, sign the manifest (minisign or cosign), and verify the signature on the appliance before loading.
• Cadence. Option A: monthly security updates through a short, logged window. Option B: monthly or quarterly bundles, tested on a reference machine first. Option C: quarterly bundles via a dedicated transfer drive, scanned on a separate machine and logged on a transfer register.

Backup, restore and ransomware recovery

Mirroring is availability, not backup. What needs protecting: the document store, the vector index (rebuildable, but slowly), the application database and configuration, user accounts and audit logs, and the encryption recovery material. Models can be re-sourced from the signed manifest.

• Snapshots (ZFS, hourly, kept for a week) give fast rollback after an accidental deletion. They are not a backup.
• Encrypted offline backups with restic or Borg to two or more rotated USB drives, at least one kept off site and disconnected. A disconnected drive cannot be encrypted by ransomware.
• The 3-2-1-1-0 rule: three copies, two media, one off site, one offline, zero errors on the last restore test.
• Bare-metal recovery is documented: reinstall from the staging image, restore the backup, re-enrol the TPM binding, and check the audit log is continuous. Test it at handover and every quarter.

Staging checklist

1. Hardware burn-in: 24 h GPU stress (gpu-burn or DCGM diagnostics level 3), memtest, NVMe SMART baseline.
2. Install Ubuntu 24.04 LTS Server, minimal; apply all updates from the snapshot; CIS Level 1 server hardening (Ubuntu Security Guide).
3. Install signed NVIDIA drivers, NVIDIA Container Toolkit and Docker; confirm nvidia-smi inside a container.
4. Install the provider's application from their checklist; load approved models; verify model hashes; apply licences.
5. Configure LUKS + Clevis/TPM (and Tang for option B), Secure Boot, firmware password and the recovery key.
6. Configure nftables, disable Wi-Fi and Bluetooth in firmware, remove unused services, disable remote-access tools.
7. Configure NUT, test a pulled-plug shutdown and the automatic restart and unlock.
8. Disconnect the network uplink: test boot, GPU detection, browser access over the staff segment, model answers, transcription and search.
9. Run the 24-hour outbound-packet log test.
10. Take the first backup, then restore it to a second drive and compare.
11. Record serials, firmware versions, package snapshot ID, container digests, model hashes and test results in a signed handover report; ship with tamper-evident seals.

Evidence and effort per security option

Option	Tests	Evidence handed over	Added effort
A. Protected network	Port scan from the staff and guest segments; outbound log test; sign-in and permission tests	Scan output, firewall ruleset, test report	Base staging
B. Controlled maintenance	As A, plus an update rehearsal from a signed bundle and an incident-isolation drill	Update runbook, transfer and change log template, Tang/TPM unlock test	About half a day more per machine, plus each update window
C. Air-gapped	As B, plus removable-media procedure walkthrough and a witnessed bare-metal restore	Transfer register, media handling procedure, restore certificate	About a day more per machine, plus backup media and a quarterly visit

Recommended builds

	Starter	Workforce	Enterprise
CPU	Ryzen 7 9700X / EPYC 4345P (8C)	Ryzen 9 9950X / EPYC 4565P (16C)	Threadripper PRO 9955WX (16C)
GPU	RTX PRO 4500 Blackwell 32 GB	RTX PRO 5000 Blackwell 48 GB	RTX PRO 6000 Blackwell Max-Q 96 GB
RAM	64 GB DDR5 ECC	128 GB DDR5 ECC	256 GB DDR5 ECC RDIMM
Storage	2× 2 TB NVMe mirror	2× 4 TB NVMe mirror	OS NVMe + 2× 4 TB NVMe mirror
UPS (230 V)	1,500 VA line-interactive	1,500 VA managed, pure sine	2,200 VA managed, pure sine
Comfortable workload	14B 8-bit or 32B 4-bit, 2–3 concurrent, with transcription	32B 4-bit, 4–6 concurrent, 16k contexts	70B 4-bit, 8–12 concurrent

The quoted VSX and Supermicro systems remain the right answer when a firm wants headroom for larger models or several practice groups on one machine, or a firm-wide server in a comms room. They are in the shop under WSL- for exactly that comparison. For an office appliance, the builds above deliver the stated workload at a much lower cost with no loss of reliability.

General technical information, not legal or regulatory advice. Benchmarks with the provider's own application and documents are required before any capacity commitment.