Turbocharging LLMs

Workstation technical brief: how to turbocharge LLM serving and agents with PagedAttention, vLLM, Self-Debugging, PowerInfer, and EG-MLA. Companion: blog · primer: LLMs on Kubernetes article · lab: Enterprise AI Lab.

1. Why serving, not training, is the crisis

Large language models changed NLP: chat, generation, tools. Training is expensive once; serving is expensive every second. Decode is autoregressive. Each new token needs the previous keys and values. Memory for that KV cache is proportional to layers × heads × hidden × sequence × batch. Prefill is compute-heavy; decode is memory-bandwidth-heavy. If you allocate a contiguous max-length KV tensor per request, most of it is empty until the sequence actually grows — classic internal fragmentation. Concurrent requests cannot steal those holes. Batch size drops. Tokens per second drop. GPUs look busy and still idle.

That is the problem PagedAttention and vLLM attacked in 2023, and the problem PowerInfer and later attention variants still attack from different angles.

Watch: LLM mental model

Context video: Intro to Large Language Models. Workstation explainer: how LLMs work and how to run them on Kubernetes.

2. PagedAttention: paging for the KV cache

Kwon, Li, Zhuang, Sheng, Zheng, Yu, Gonzalez, Zhang, and Stoica introduced PagedAttention as an attention algorithm inspired by virtual memory and OS paging, and built vLLM on top of it [arXiv:2309.06180]. The idea:

• Split KV into fixed-size blocks (a small number of tokens per block).
• Keep a block table from logical token positions to physical GPU blocks (possibly non-contiguous).
• Allocate/free blocks as sequences grow or finish — like page allocation, not like malloc of one giant array.
• Share blocks (copy-on-write) for prefix reuse, beam search, and parallel sampling so you do not duplicate identical prefixes.

Implementation is not “set a flag on BERT.” The attention kernel must gather scattered blocks. The scheduler must know which blocks are free. The cache hierarchy matters: HBM-resident blocks vs CPU offload vs NVLink peers. Buffer (block) size is a real knob. Too small: more table lookups and kernel overhead. Too large: wasted slots inside the last partial block. Pair block size with max_model_len and the actual prompt/completion mix of your traffic.

Practice: profile fragmentation (unused KV bytes / reserved KV bytes) and tokens/s together. Memory graphs without throughput are vanity.

3. vLLM: the serving system around the pager

vLLM’s claim is near-zero waste in KV cache memory plus flexible sharing within and across requests. Evaluations in the paper showed roughly 2–4× throughput versus then-SOTA systems (FasterTransformer, Orca) at similar latency, with larger gains on long sequences and fancier decoding. Today the engine also ships continuous batching, chunked prefill, prefix caching, and tensor/pipeline parallel for multi-GPU.

Operational checklist:

1. Set gpu_memory_utilization high enough to hold weights + KV, low enough to leave CUDA workspace.
2. Enable prefix caching only after you confirm eval scores and p95 TTFT on your prompts.
3. Separate prefill-heavy and decode-heavy pools if mixed traffic wrecks SLO (disaggregated serving).
4. Expose OpenAI-compatible endpoints behind your gateway (Workstation estates often put this behind WSL Proxy / API gateway patterns).

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    gpu_memory_utilization=0.90,
    max_model_len=8192,
    enable_prefix_caching=True,
)
params = SamplingParams(temperature=0.2, max_tokens=256)
outs = llm.generate(["Summarise PagedAttention for a platform engineer."], params)
print(outs[0].outputs[0].text)

Anti-pattern (this is not PagedAttention):

# Classifier forward pass — logits, not paged KV serving
import torch
from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased")
out = model(torch.tensor([[1, 2, 3]]), attention_mask=torch.tensor([[1, 1, 1]]))
print(out.logits)

Watch: serving systems in the wild

Contextual serving talk. Canonical write-up: vLLM blog (PagedAttention) · engine: vllm-project/vllm.

4. Self-Debugging: more quality per candidate, not more candidates

Chen, Lin, Klein, et al. showed that teaching an LLM to debug its predicted program with few-shot demonstrations can match or beat baselines that generate more than 10× as many candidates [arXiv:2304.05128]. The loop is: generate → execute or unit-test → feed traces back → repair.

Production trade-off: each feedback message is another prefill+decode (or a long context append). Accuracy often rises with more rounds; so does latency and cost. Workstation guidance for agents (see also Muse Glimmer / local agents):

• Hard cap on debug rounds (for example 2–4) per tool call.
• Budget tokens separately from user-visible chat.
• Escalate to a human or a specialist model instead of infinite retry.
• Log traces for eval — Self-Debugging without telemetry is folklore.

5. PowerInfer: locality-aware token generation

PowerInfer (SJTU IPADS / related repos) is a token generation system that exploits activation locality: a small “hot” neuron set on GPU, colder weights streamed or executed on CPU. Published operating points include 13.20 tokens/s average and 29.08 tokens/s peak on a single NVIDIA RTX 4090, and up to 11.69× versus llama.cpp with retained accuracy [PowerInfer GitHub]. (User-facing forks such as Tiiny-AI/PowerInfer track the same line of work.)

Scale-out is the hard part. A single 4090 locality profile does not automatically become a healthy Kubernetes Deployment. You need:

• Honest GPU requests/limits and NUMA-aware CPU pinning if CPU experts run.
• A distributed plan if the model no longer fits: tensor parallel vs pipeline vs expert parallel.
• SLO split: interactive chat vs batch completion vs agent tool loops.

Use PowerInfer (or llama.cpp, or MLX) on workstations and edge boxes; use vLLM (or TensorRT-LLM, or SGLang) when you are filling datacenter GPUs with concurrent OpenAI-style traffic. Measure both. Our Enterprise AI Lab stance is the same as Polyglot Benchmarks: evidence, then ADR.

6. EG-MLA: compress attention at the architecture

Serving tricks cannot fix a model whose KV is intrinsically huge. EG-MLA (embedding-gated multi-head latent attention) reports over 91.6% KV cache size reduction versus multi-head attention (MHA) with negligible degradation, additional savings versus MLA (up to 59.9%), and improved reasoning-benchmark accuracy. The authors argue embedding gating induces implicit high-order interactions and show scaling past 1B parameters [EG-MLA, arXiv].

Engineering implication: this is a training / architecture decision. You cannot flip EG-MLA on a random vLLM nightly of Llama-3 and expect the paper’s percentages. If you control pretrain or continued pretrain, EG-MLA is a candidate to cut HBM before you buy another GPU. If you only serve public weights, stay on PagedAttention + quantization + MLA variants the engine already supports, and track EG-MLA checkpoints as they land.

7. Combining the stack without cargo-culting

Layer	Use when	Watch out
PagedAttention / vLLM	Concurrent API serving, long context, shared prefixes	Prefix cache vs correctness; OOM at high util
PowerInfer	Single fat GPU / workstation token rate	Locality mismatch; messy scale-out
Self-Debugging	Code / agent loops that can execute tests	Unbounded rounds; extra prefill cost
EG-MLA	You train or fine-tune the backbone	Not a serving flag; re-run full eval

8. Scalability on Kubernetes

A well-designed distributed architecture raises throughput and also raises failure modes: stragglers, KV cache transfer, tokenizer skew, and autoscalers that kill warm prefixes. Practical pattern (aligned with our Ollama / vLLM on Kubernetes write-up):

• Dedicated GPU node pools; never pack decode pods with random CPU jobs.
• Separate prefill and decode if TTFT SLOs and tokens/s SLOs fight.
• HPA on queue depth or GPU KV occupancy, not only CPU.
• Promote serving stacks through rings (Ring Promoter) so a bad engine build cannot skip test.

9. Conclusion

Turbocharging LLMs is not one algorithm. It is paging the KV cache (PagedAttention), a serving engine that does not waste those pages (vLLM), locality-aware generation when the hardware is a workstation GPU (PowerInfer), agent loops that debug instead of spraying candidates (Self-Debugging), and — when you own the weights — attention that simply stores less (EG-MLA). Every layer trades memory, latency, and accuracy. Measure on your traffic. Publish the ADR. Ship.

References

• Kwon et al. — Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv:2309.06180, 14 Sep 2023).
• Chen et al. — Teaching Large Language Models to Self-Debug (arXiv:2304.05128, 12 Apr 2023).
• PowerInfer — SJTU-IPADS/PowerInfer (and related Tiiny-AI forks).
• EG-MLA — Embedding-Gated Multi-head Latent Attention (arXiv, 20 Sep 2025).
• vLLM blog — Easy, Fast, and Cheap LLM Serving with PagedAttention.

Published by Workstation. Paper figures cited as published by the original authors; production numbers on your cluster will differ.