[Eight Gigs]
Local LLM Lab
A 9B model with 32k context entirely on an 8 GB laptop GPU, and a 27B model on a rented 48 GB GPU — behind the same OpenAI-compatible API.
The agent graph makes dozens of LLM calls per asset. I wanted to develop against a model at home, run full batches on a bigger model in the cloud, and never touch agent code to switch between them.
Solo: model and quantization choices, server scripts and the serverless GPU deployment.
By default an 8 GB GPU doesn't hold a 9B model plus a long context, and the first cloud GPU I picked ran out of memory with the 27B model.
01 · Quantized GGUF on llama.cpp
A 9B model tested at Q4_K_M, Q5 and Q6, plus a smaller model at Q6. Q4_K_M is the one that leaves room for a 32k context.
02 · Every layer on the GPU
-ngl 99 puts all 33 layers on the GPU. That only fit after moving the desktop display to the integrated GPU; before that, 4 layers spilled to system RAM.
03 · Two server profiles
One start script with reasoning on (for manager agents) and one with it off (for fast analyst calls).
04 · 27B AWQ on vLLM
A 4-bit AWQ build served by vLLM on a serverless GPU: API key from the platform's secret store, Qwen reasoning and tool-call parsers, weights on a persistent volume, scale-to-zero after 15 idle minutes.
05 · One API everywhere
Agents only see an OpenAI-compatible URL; local, self-hosted and hosted providers are a config switch.
what you're seeing
The bar is GPU memory: weights (cyan), KV cache (lime), runtime overhead (grey) and the budget line (amber). Below, each square is a transformer layer — on the GPU or spilled to system RAM, which is what makes generation crawl.try this
- 1.Start at 9B · Q4_K_M · 8 GB laptop with 32k context: it doesn't fit while the display uses the GPU.
- 2.Turn on “move display to iGPU” — the last layers come back to the GPU.
- 3.Switch to 27B · AWQ on 24 GB, then on 48 GB, and shrink the context to 16k.
real vs simplified
The scenarios match my setup (9B with 33 layers on an 8 GB laptop GPU; a 27B AWQ build that ran out of memory on 24 GB and runs on 48 GB). Memory and speed formulas are rough estimates, not a benchmark.! brokeA 24 GB GPU ran out of memory: the 27B weights alone take about 21 GB.
→ changedMoved to a 48 GB GPU.
! brokeEven on 48 GB, the AWQ build left little room for the KV cache at 32k context.
→ changedCut the max context to 16k and enforced eager mode, trading CUDA-graph speed for memory.
! brokeThe pinned vLLM version predated the Qwen reasoning parser, and the slim base image had no CUDA compiler for kernel JIT.
→ changedUnpinned vLLM and switched to a CUDA devel base image.