No products in the cart.
How Much VRAM Does Agentic AI Actually Need? The KV Cache Math
Here is the number that surprises most teams: a 70B model quantized to FP8 fits comfortably on a single 96GB GPU, with room left over for zero concurrent agent sessions at 128K context.
The model loads. The demo works. Then you point three agents at it in production and the serving framework starts evicting context mid-task. This is the most common way an agentic AI deployment fails, and it happens because almost every sizing guide stops at parameter count.
This post gives you the actual arithmetic. Not rules of thumb — the formula, worked through for real models at real context lengths, plus a table showing exactly how many concurrent agent sessions fit on a 96GB card. Everything here you can recompute yourself from a model’s config.json.
VRAM has two parts, and only one of them is obvious
Total GPU memory for inference breaks into three pieces:
Total VRAM = model weights + KV cache + framework overhead
Weights are simple: parameter count multiplied by bytes per parameter. BF16/FP16 is 2 bytes, FP8/INT8 is about 1 byte, INT4 is about 0.5 bytes. A 70B model is 140GB at BF16, 70GB at FP8, 35GB at INT4.
Framework overhead — CUDA context, activations, allocator fragmentation — runs roughly 10 to 20 percent on top of weights. We use 15 percent throughout this post.
KV cache is the part that breaks deployments. It stores the attention keys and values for every token already in the context so the model doesn’t recompute them on each new token. It scales linearly with context length and linearly with the number of concurrent sessions. For agents, both of those numbers are much larger than for chat.
The KV cache formula
KV bytes/token = 2 × layers × KV_heads × head_dim × bytes_per_element
Session KV = KV bytes/token × context_length
Total KV = Session KV × concurrent_sessions
The leading 2 accounts for one key tensor and one value tensor. Every other term except context length is a fixed architectural constant you can read straight out of the model’s config.json: num_hidden_layers, num_key_value_heads, and head_dim. The last term is 2 bytes for a BF16 cache or 1 byte for FP8.
num_key_value_heads is the term people get wrong. Modern models use grouped-query attention, where many query heads share one KV head. Llama 3.3 70B has 64 query heads but only 8 KV heads — an 8x reduction in cache size. Use the query head count by mistake and your estimate is off by nearly an order of magnitude.
Worked for Llama 3.3 70B (80 layers, 8 KV heads, 128 head dimension, BF16 cache):
2 × 80 × 8 × 128 × 2 = 327,680 bytes/token ≈ 320 KiB
× 131,072 tokens (128K context) = 40 GB per session
× 131,072 tokens (128K context) = 40 GB per session
Forty gigabytes. For one agent, holding one conversation.
KV cache by model and context length
Same arithmetic applied to three models teams actually deploy for agent work. Architecture values are from each model’s published configuration.
| Model | Layers / KV heads | KV precision | 32K | 128K | 256K |
|---|---|---|---|---|---|
| Llama 3.1 8B | 32 / 8 | BF16 | 4.0 GB | 16.0 GB | 32.0 GB |
| Llama 3.1 8B | 32 / 8 | FP8 | 2.0 GB | 8.0 GB | 16.0 GB |
| Qwen3-32B | 64 / 8 | BF16 | 8.0 GB | 32.0 GB | 64.0 GB |
| Qwen3-32B | 64 / 8 | FP8 | 4.0 GB | 16.0 GB | 32.0 GB |
| Llama 3.3 70B | 80 / 8 | BF16 | 10.0 GB | 40.0 GB | 80.0 GB |
| Llama 3.3 70B | 80 / 8 | FP8 | 5.0 GB | 20.0 GB | 40.0 GB |
Two things to notice. A 70B at 256K context needs 80GB of KV cache alone — nearly a full 96GB card before you load a single weight. And halving the cache precision from BF16 to FP8 halves every number in the table, which makes KV quantization the single highest-leverage change available to you.
One caveat worth knowing: not every model ships with a native 128K window. Qwen3-32B is trained at roughly 41K and reaches 128K through RoPE scaling. Check the model card before you budget for context the model was never trained to use well.
The reality check: concurrent agents on one 96GB GPU
This is the table that matters. Each row assumes a 96GB GPU, weights plus 15 percent overhead loaded, and the remainder divided by one 128K-token session.
| Model | Weights | KV precision | Weights + overhead | Free for KV | Per session | Concurrent sessions |
|---|---|---|---|---|---|---|
| Llama 3.1 8B | FP8 | FP8 | 9.2 GB | 86.8 GB | 8.0 GB | 10 |
| Llama 3.1 8B | FP8 | BF16 | 9.2 GB | 86.8 GB | 16.0 GB | 5 |
| Qwen3-32B | INT4 | FP8 | 18.4 GB | 77.6 GB | 16.0 GB | 4 |
| Qwen3-32B | FP8 | FP8 | 36.8 GB | 59.2 GB | 16.0 GB | 3 |
| Qwen3-32B | FP8 | BF16 | 36.8 GB | 59.2 GB | 32.0 GB | 1 |
| Llama 3.3 70B | INT4 | FP8 | 40.2 GB | 55.8 GB | 20.0 GB | 2 |
| Llama 3.3 70B | INT4 | BF16 | 40.2 GB | 55.8 GB | 40.0 GB | 1 |
| Llama 3.3 70B | FP8 | FP8 | 80.5 GB | 15.5 GB | 20.0 GB | 0 |
That last row is the failure mode in one line. A 70B at FP8 weights loads perfectly on a 96GB card — 80.5GB with overhead, 15.5GB to spare. It will pass every smoke test you throw at it. It cannot serve a single 128K agent session, because that session needs 20GB and you have 15.5.
Nobody discovers this in development, where sessions are short. Everybody discovers it in week two of production, when an agent’s tool-call history grows past the point where the numbers still work.
Four gigabytes of RTX PRO 6000 VRAM is not 384GB of usable pool
The RTX PRO 6000 Blackwell has no NVLink. Multi-GPU traffic runs over PCIe Gen 5, so a four-card workstation is four independent 96GB pools, not one 384GB pool. Every number in the table above applies per card.
This sounds like a limitation and is mostly an advantage for agent work. Agentic pipelines are naturally heterogeneous: a small router, one or two mid-size task models, a validator, maybe a larger synthesizer. Four separate pools map onto that shape better than one shared pool does, and they give you something a single pool cannot — isolation. A runaway 500K-token session on your research agent cannot starve your router of cache, because they are not sharing memory.
A representative four-role allocation on a quad-GPU workstation:
| GPU | Agent role | Model class | Context budget |
|---|---|---|---|
| GPU 0 | Router / classifier | 8B at FP8 | 10 sessions @ 128K |
| GPU 1 | Coding agent | 32B at FP8 | 3 sessions @ 128K |
| GPU 2 | Retrieval synthesis | 32B at INT4 | 4 sessions @ 128K |
| GPU 3 | Reasoning / planner | 70B at INT4 | 2 sessions @ 128K |
Tensor-parallel splitting one model across two cards is possible when a model genuinely exceeds 96GB, but PCIe adds latency on every layer boundary. Use it when you must, not by default. For the hardware side of quad-GPU builds, see our 4x RTX PRO 6000 thermal validation and the RTX PRO 6000 edition guide.
Four levers when the math doesn’t fit
In order of return on effort:
- Quantize the KV cache to FP8. Halves cache memory, doubles either your context ceiling or your session count. Costs you nothing in weights and very little in quality for most agent tasks. Validate on your own evals first.
- Quantize the weights. FP8 to INT4 on a 70B frees 35GB — which buys nearly two more 128K sessions. Quality loss is real but usually acceptable for routing, extraction, and tool-calling roles.
- Cap the context. Most agents don’t need 128K. Cap at 32K and a 70B at INT4 with FP8 KV goes from 2 sessions to 11. Aggressive context management — summarizing prior steps instead of carrying them verbatim — is often the cheapest fix available.
- Add GPUs. When roles genuinely need to run in parallel at long context, more independent pools is the answer. This is where a 4-GPU workstation earns its cost.
Note what is not on this list: buying a larger model. Capacity determines whether a model loads; bandwidth determines how fast it generates once loaded. Neither is fixed by adding parameters.
Sizing your build from the numbers
Work backward from three inputs — model size, peak context, peak concurrency — and the hardware follows.
| Your workload | Configuration | Build |
|---|---|---|
| 1-2 agents, moderate context, single developer | 1x RTX PRO 6000 (96GB) | Ryzen · Intel Xeon |
| 2-3 parallel roles, long context | 2x RTX PRO 6000 Max-Q (192GB) | Threadripper PRO |
| Full multi-agent stack, 4 roles in parallel | 4x RTX PRO 6000 Max-Q (4 x 96GB) | Threadripper PRO tower · 5U rack |
| Shared team deployment, high concurrency | 4-8x RTX PRO 6000 Server Edition | 4U EPYC server |
VRLA Tech at vrlatech.com sizes builds from this arithmetic before quoting. Tell us the models you are serving, your peak context, and how many agents run concurrently, and an engineer will work the numbers with you rather than pointing you at the largest system on the page. Every build ships burn-in tested with vLLM, SGLang, PyTorch, and CUDA pre-configured. Most workstations and servers ship within 2-3 weeks, with a 3-year parts warranty and lifetime US-based engineer support. Built in Los Angeles since 2016. Customers include General Dynamics, Los Alamos National Laboratory, Johns Hopkins University, Miami University, and George Washington University.
Ready to buy?
KV cache and sizing questions
- How do you calculate KV cache size?
- KV bytes per token = 2 × layers × KV heads × head dimension × bytes per element. Multiply by context length for one session, then by concurrent sessions. For Llama 3.3 70B: 2 × 80 × 8 × 128 × 2 = 327,680 bytes per token, or 40GB at 128K context in BF16. All four values come from the model’s config.json. VRLA Tech sizes GPU memory from this arithmetic before quoting a build. Los Angeles since 2016, 3-year parts warranty, lifetime US-based engineer support.
- How much VRAM do I need to run a 70B model as an agent?
- More than the weights suggest. A 70B at FP8 is roughly 70GB of weights plus 15% overhead, and each 128K-token session adds 40GB of BF16 KV cache. On a single 96GB GPU that leaves room for zero concurrent long-context sessions. At INT4 weights and FP8 KV you fit about two. VRLA Tech builds multi-GPU workstations sized to concurrency, not parameter count. Built in Los Angeles since 2016 with a 3-year parts warranty and lifetime US-based engineer support.
- Is a 4x RTX PRO 6000 workstation 384GB of usable VRAM?
- Not as one pool. RTX PRO 6000 Blackwell has no NVLink, so multi-GPU traffic runs over PCIe Gen 5 and the four cards behave as four independent 96GB pools. That is excellent for running four different agent roles in parallel, and poor for one model that needs more than 96GB. VRLA Tech builds 4-GPU workstations in Los Angeles. Since 2016, 3-year parts warranty, lifetime US-based engineer support.
- Why does agentic AI need more memory than a chatbot?
- An agent carries system instructions, tool schemas, retrieved documents, and every prior step of the task forward in its context. A chat turn that runs 2,000 tokens becomes 100,000 tokens in an agent loop, and KV cache scales linearly with that. Then every concurrent agent pays the cost again. VRLA Tech sizes for peak context and concurrency rather than parameter count. Built in Los Angeles since 2016. 3-year parts warranty and lifetime US-based engineer support. Customers include General Dynamics and Los Alamos National Laboratory.
- Does FP8 KV cache quantization hurt agent quality?
- FP8 KV cache halves cache memory and is the highest-leverage change available, with quality degradation that is small for most agent workloads. It doubles either your context ceiling or your concurrent session count on the same card. Validate on your own evaluation set before production, since sensitivity varies by model and task. VRLA Tech pre-configures vLLM and SGLang with KV quantization on every build. Los Angeles since 2016, 3-year parts warranty, lifetime US-based engineer support.
- What happens when an agent runs out of KV cache mid-task?
- The serving framework either fails the request with an out-of-memory error or silently evicts earlier context to make room. For an agent that depends on its own history, silent eviction is the worse outcome: it keeps running and quietly forgets what it already did. Sizing for peak context and peak concurrency up front is the only reliable fix. VRLA Tech validates memory headroom before shipping. Built in Los Angeles since 2016. 3-year parts warranty and lifetime US-based engineer support.
Hardware and buying questions
- How many AI agents can one RTX PRO 6000 run at once?
- At 128K context on one 96GB card: roughly 10 sessions on an 8B at FP8 weights and FP8 KV, 3 to 4 on a 32B, and 0 to 2 on a 70B depending on quantization. Shorter contexts raise all of these substantially. VRLA Tech sizes GPU count from your model, context, and concurrency targets. Los Angeles since 2016, 3-year parts warranty, lifetime US-based engineer support.
- Should I buy one big GPU or several smaller ones for agents?
- For multi-agent stacks running distinct roles, several cards win: each GPU hosts its own model with its own isolated KV cache, and a runaway session on one agent cannot starve the others. One large pool only wins when a single model genuinely exceeds one card. VRLA Tech builds both. Built in Los Angeles since 2016. 3-year parts warranty and lifetime US-based engineer support. Customers include Johns Hopkins University.
- What GPU should I buy for local agentic AI?
- Work backward from concurrency. One or two agents at moderate context fit a single RTX PRO 6000 Blackwell. Four parallel agent roles at long context want a 4-GPU Max-Q workstation, giving each role its own 96GB pool. Beyond that, move to a rackmount GPU server. VRLA Tech builds all three tiers in Los Angeles. Since 2016, 3-year parts warranty, lifetime US-based engineer support. Customers include General Dynamics and George Washington University.
- Do I need a GPU server instead of a workstation for agents?
- Move to a server when concurrency outgrows four GPUs, when several teams share the deployment, or when the stack needs remote management and redundant power. A workstation is the better fit for one team iterating at the desk. VRLA Tech builds both, including 4U rackmount GPU servers. Built in Los Angeles since 2016. 3-year parts warranty and lifetime US-based engineer support.
- Where can I buy an agentic AI workstation?
- VRLA Tech builds agentic AI workstations and GPU servers to order in Los Angeles, sized from your model, context length, and concurrency rather than from a spec sheet. Every system ships burn-in tested with vLLM, SGLang, PyTorch, and CUDA pre-configured. Most workstations and servers ship within 2-3 weeks. 3-year parts warranty and lifetime US-based engineer support. Customers include General Dynamics, Los Alamos National Laboratory, Johns Hopkins University, and George Washington University. In business since 2016.
For framework-level guidance on running agents locally, see Best Workstation for Local AI Agents. For the quad-GPU hardware validation behind these builds, see our 4x RTX PRO 6000 thermal test. For production serving architecture, see the AI Inference Server Configuration Guide. To model on-premise versus cloud cost, use the AI ROI Calculator.
VRLA Tech builds for defense and government, healthcare, research laboratories, finance, and pharmaceutical and biotech organizations.
