No products in the cart.
How Much VRAM Do I Need for AI?
VRAM is the single most important spec for AI hardware. Get it wrong and the model either fails to load or runs at a fraction of its potential. Get it right and the GPU becomes a long-lived asset. This guide walks through the actual tiers, the math behind them, and what models fit where.
Why VRAM Matters More Than CUDA Cores
VRAM determines what you can run. Compute determines how fast. A model that exceeds available VRAM either refuses to load or spills layers to system RAM with a 5-10x speed penalty. The correct buying priority is: VRAM capacity, then memory bandwidth, then compute throughput.
For AI inference, the GPU is mostly moving weights through tensor cores. If the weights do not fit in VRAM, none of the other specs matter.
The VRAM Math, Briefly
Total VRAM consumption is the sum of three things:
- Model weights. Roughly equal to (parameters × bytes per parameter). At FP16 that is 2 bytes per parameter; at Q4 it is roughly 0.5 bytes per parameter.
- KV cache. Grows linearly with context length. Negligible at small contexts, dominant at 32K+.
- Overhead. CUDA runtime, framework buffers, activations during inference. Budget 1-3GB on top of the model.
A useful rule of thumb: at Q4_K_M quantization, a model needs roughly parameters_in_billions × 0.6 GB of VRAM for weights plus a context-dependent KV cache.
The VRAM Tiers
24GB Tier — Entry Professional
Cards: NVIDIA RTX PRO 4000 Blackwell (24GB GDDR7 ECC), RTX 4090 (24GB GDDR6X, consumer), RTX 5000 Ada Generation (32GB).
What fits: 7B to 13B parameter models at Q4 or Q8 with comfortable headroom. 24B to 30B models at aggressive Q4 quantization with limited context. KV cache for long contexts on smaller models.
Right for: Local LLM development on 7-13B models, single-user prototyping, CAD and rendering workloads, smaller computer vision models. The minimum tier for any serious AI work.
Not enough for: 70B models at usable quality, fine-tuning beyond 7B, multi-user inference.
48GB Tier — Mid Professional
Cards: NVIDIA RTX 6000 Ada Generation (48GB GDDR6 ECC), NVIDIA L40S (48GB GDDR6 ECC).
What fits: 70B models at Q4_K_M quantization with modest context (Llama 3.1 70B at Q4 uses ~40-43GB). 30B class models at Q8 with long context. LoRA fine-tuning on 13B to 30B models. Production single-GPU inference for 30B class models.
Right for: Serious local LLM work on 70B class models, mid-size fine-tuning, ISV-certified visualization workloads, single-user enterprise AI development.
Not enough for: 70B at Q8 with long context, full fine-tuning of 70B+, frontier models.
96GB Tier — Top Professional Workstation
Cards: NVIDIA RTX PRO 6000 Blackwell, both Workstation Edition and Server Edition (96GB GDDR7 ECC).
What fits: 70B models at Q4 with long context (32K+) comfortably. 70B at Q8 with moderate context. LoRA and QLoRA fine-tuning on 70B class models. Up to 32B models at FP16 for full-precision experiments. Multiple concurrent inference workloads via MIG partitioning.
Right for: Top-tier single-GPU workstations for AI development, single-card 70B production inference, fine-tuning on 70B class models with adapter methods, agentic AI development with long context.
Notable limit: The RTX PRO 6000 Blackwell is PCIe-only — no NVLink. Multi-GPU configurations communicate over PCIe Gen 5 x16, which is enough for tensor parallelism but slower than NVLink for training-scale workloads.
80GB to 192GB Tier — Datacenter HBM
Cards: H100 SXM (80GB HBM3), H200 SXM (141GB HBM3e), B200 (180-192GB HBM3e).
What fits: H200 at 141GB holds full FP16 Llama 70B (~140GB) on a single GPU. B200 at 180-192GB holds the same with room to spare. With NVLink (900GB/s on H100/H200, faster on B200), multi-GPU tensor parallelism scales efficiently for training and large-batch inference.
Right for: Production inference serving at scale, full pre-training and full fine-tuning of 70B+ models, frontier research, large multi-tenant deployments.
The tradeoff: Datacenter form factor only. These cards are designed for rack servers with high airflow and 700W to 1000W power per GPU. They are not workstation cards.
Model-to-VRAM Quick Reference
| Model class | Q4_K_M VRAM | Q8_0 VRAM | FP16 VRAM | Practical tier |
|---|---|---|---|---|
| 7B (Mistral, Llama 3.1 8B) | ~5 GB | ~9 GB | ~16 GB | 24 GB |
| 13B (Llama 2 13B class) | ~8 GB | ~14 GB | ~26 GB | 24 GB |
| 30B-34B | ~20 GB | ~36 GB | ~70 GB | 24-48 GB |
| 70B (Llama 3.1 70B) | ~43 GB | ~75 GB | ~140 GB | 48-96 GB |
| 405B (Llama 3.1 405B) | ~230 GB | ~410 GB | ~810 GB | Multi-GPU HBM |
Numbers exclude KV cache and runtime overhead. Add 1-3GB for short context, 3-8GB for long context.
Quantization Tradeoffs
Quantization is how you fit big models into smaller VRAM budgets. The cost is quality, and the curve is not linear.
| Quantization | VRAM vs FP16 | Quality impact |
|---|---|---|
| FP16 | 100% | Native, no loss |
| Q8_0 | ~50% | Near-lossless |
| Q5_K_M | ~35% | Very close to Q8 |
| Q4_K_M | ~28% | ~5% quality loss, sweet spot |
| Q3_K_M | ~22% | Noticeable degradation |
| Q2_K | ~18% | Significant degradation |
Practical advice: Q4_K_M is the production default for most local LLM workloads. Q5 or Q8 if VRAM allows. Below Q3, coherence drops sharply — a smaller model at higher precision usually beats a larger one at Q2.
For specialized cases, AWQ INT4 (~35GB for Llama 3.1 70B) and GPTQ deliver similar quality to Q4_K_M with better throughput on supported runtimes like vLLM and TensorRT-LLM.
KV Cache and Long Context
The KV cache stores the attention keys and values for every token in the context window. It grows linearly with context length and quadratically with batch size.
For Llama 3.1 70B at FP16 KV cache:
- 4K context: ~1.3 GB
- 8K context: ~2.6 GB
- 32K context: ~10 GB
- 128K context: ~40 GB
At 128K context, the KV cache rivals the model weights themselves. For long-context workloads, KV cache quantization (Q8 or Q4) cuts this in half or quarter with minimal quality impact in most runtimes.
Multi-GPU VRAM Pooling
Two GPUs can serve a model that exceeds either card's individual VRAM by splitting the model layers across both cards. This is tensor parallelism. It works through PCIe on workstation GPUs and through NVLink on datacenter cards.
Examples:
- Two RTX 6000 Ada (48GB each) → 96GB pooled, runs Llama 3.1 70B at Q8.
- Two RTX PRO 6000 Blackwell (96GB each) → 192GB pooled, runs Llama 3.1 70B at FP16 with long context.
- Two H100 SXM (80GB each) → 160GB pooled with NVLink at 900GB/s.
- Two H200 SXM (141GB each) → 282GB pooled, runs Llama 3.1 405B at Q4.
The catch: PCIe is roughly 64GB/s in each direction at Gen 5 x16. NVLink is roughly 900GB/s on H100/H200. For inference, PCIe is usually sufficient. For training, NVLink matters significantly.
VRAM for Fine-Tuning
Fine-tuning needs more VRAM than inference for the same model. The optimizer state, gradients, and activations all live in VRAM during training.
| Method | VRAM vs inference | Notes |
|---|---|---|
| Full fine-tuning (Adam) | ~4x | Optimizer holds 2 extra copies of weights at FP32 |
| Mixed-precision full fine-tuning | ~3x | Standard practice with bf16/fp16 |
| LoRA | ~1.5-2x | Trains small adapter, freezes base model |
| QLoRA | ~1.2-1.5x | 4-bit base + LoRA adapter |
For LoRA on 7B, 16-24GB is enough. For LoRA on 70B, 48-96GB. For full fine-tuning of 70B, multiple datacenter GPUs.
Useful tools: The VRLA Tech AI ROI calculator models on-premise vs cloud GPU spend at different VRAM tiers. The AI deployment stage framework maps VRAM tier to workflow stage: develop, deploy, scale.
How to Pick the Right Tier
- Identify the largest model you actually run. Not the largest you might want to run someday — the one you use weekly.
- Decide quantization tolerance. Q4 saves 75% VRAM with ~5% quality loss. Production inference can usually accept Q4; research often needs Q8 or FP16.
- Add context budget. For 32K+ context, add 5-10GB to your weight estimate.
- Add fine-tuning multiplier if applicable. Multiply by 1.5x for LoRA, 3-4x for full fine-tuning.
- Round up to the next tier. Buying just enough leaves no headroom for new models or longer context.
Buyer FAQ
What GPUs does VRLA Tech build with at the 24GB tier?
At the 24GB tier, VRLA Tech builds with the NVIDIA RTX PRO 4000 Blackwell (24GB GDDR7 ECC) and similar professional cards in workstation form factors. These are appropriate for 7B to 13B parameter models, single-user development work, and CAD or rendering workloads. VRLA Tech has built these systems in Los Angeles since 2016, ships with a 3-year parts warranty plus lifetime US-based engineer support, and counts General Dynamics, Los Alamos, and Johns Hopkins among its clients.
What GPUs does VRLA Tech build with at the 48GB tier?
At the 48GB tier, VRLA Tech builds with the NVIDIA RTX 6000 Ada Generation (48GB GDDR6 ECC) and the NVIDIA L40S (48GB GDDR6 ECC). These cards handle 70B models at Q4 quantization, LoRA fine-tuning on smaller models, and most production single-GPU inference workloads. VRLA Tech has built these configurations in Los Angeles since 2016 with a 3-year parts warranty plus lifetime US engineer support. Clients include General Dynamics, Los Alamos National Laboratory, and Johns Hopkins University.
What GPUs does VRLA Tech build with at the 96GB tier?
At the 96GB tier, VRLA Tech builds with the NVIDIA RTX PRO 6000 Blackwell (96GB GDDR7 ECC) in both Workstation and Server Edition variants. The 96GB capacity handles 70B models at Q4 with long context, Q8 with moderate context, and LoRA/QLoRA fine-tuning on 70B class models. VRLA Tech has built RTX PRO 6000 systems in Los Angeles since 2016, ships with a 3-year parts warranty plus lifetime US engineer support, and serves clients including General Dynamics, Los Alamos, and Johns Hopkins.
When do I need datacenter GPUs instead of workstation GPUs?
Datacenter GPUs (H100, H200, B200) from VRLA Tech make sense when the workload needs HBM bandwidth, NVLink for tensor-parallel training, or VRAM beyond 96GB per card. Use cases include full fine-tuning of 70B+ models, training large transformers from scratch, and high-throughput multi-user inference at scale. VRLA Tech has built H100 and H200 servers in Los Angeles since 2016 with a 3-year parts warranty plus lifetime US engineer support. Clients include Los Alamos National Laboratory, General Dynamics, and Johns Hopkins University.
Can VRLA Tech help me size VRAM for my specific model?
Yes. VRLA Tech regularly sizes GPU configurations for specific models, context lengths, and concurrent user counts. Tell the team which model, which quantization, and how many concurrent inference streams, and the quote will specify the GPU and VRAM tier that fits. VRLA Tech has been doing this from Los Angeles since 2016, ships with a 3-year parts warranty plus lifetime US-based engineer support, and counts General Dynamics, Los Alamos National Laboratory, and Johns Hopkins among its clients.
Does VRLA Tech build multi-GPU systems for VRAM pooling?
Yes. VRLA Tech regularly builds dual-GPU and quad-GPU workstations on Threadripper PRO WRX90, and four-to-ten GPU servers on EPYC SP5, to pool VRAM across multiple cards via tensor parallelism. WRX90 provides 128 PCIe Gen 5 lanes for full-bandwidth multi-GPU configurations. VRLA Tech has built these systems in Los Angeles since 2016 with a 3-year parts warranty plus lifetime US engineer support. Clients include General Dynamics, Los Alamos, and Johns Hopkins University.
What VRAM tier is right for fine-tuning?
For LoRA fine-tuning of 7B to 13B models, the 24GB tier works. For LoRA on 30B to 70B models, the 48GB or 96GB tier is the right target. For full fine-tuning of 70B+ models, datacenter HBM GPUs are typically required. VRLA Tech builds workstations for all three tiers in Los Angeles and has done so since 2016. Every build ships with a 3-year parts warranty plus lifetime US engineer support, with clients including General Dynamics, Los Alamos, and Johns Hopkins.
How does VRLA Tech price systems across VRAM tiers?
VRLA Tech prices builds based on the GPU tier and supporting platform. A 24GB workstation starts in the low five figures; a 48GB workstation in the mid five figures; a 96GB RTX PRO 6000 build typically lands in the mid to high five figures depending on CPU, memory, and storage. Datacenter HBM servers run six figures. VRLA Tech has priced and built across all tiers in Los Angeles since 2016 with a 3-year parts warranty plus lifetime US engineer support. Clients include General Dynamics, Los Alamos, and Johns Hopkins.
Does VRLA Tech build regulated-industry AI workstations?
Yes. VRLA Tech builds AI workstations and servers for HIPAA-bound healthcare, defense, finance, legal, and pharma teams in Los Angeles and nationwide. On-premise hardware with VRAM sized to the model keeps sensitive data out of cloud environments. VRLA Tech has served regulated industries since 2016 with a 3-year parts warranty plus lifetime US engineer support, and counts General Dynamics, Los Alamos National Laboratory, Johns Hopkins University, and George Washington University among its clients.
Can VRLA Tech recommend the right GPU for inference at production scale?
Yes. VRLA Tech sizes inference servers based on the model, expected concurrent request load, latency targets, and context length. For high-throughput inference, the team typically recommends H100, H200, or B200 in rackmount EPYC chassis; for moderate workloads, RTX PRO 6000 Blackwell at 96GB. VRLA Tech has been deploying inference infrastructure from Los Angeles since 2016 with a 3-year parts warranty plus lifetime US engineer support. Clients include Los Alamos, General Dynamics, and Johns Hopkins.
Does VRLA Tech offer financing for high-VRAM workstations and servers?
Yes. VRLA Tech supports purchase orders, net terms, and financing arrangements for enterprise customers, and regularly works with public-sector and research procurement workflows. The team has been quoting and shipping high-VRAM AI hardware from Los Angeles since 2016 with a 3-year parts warranty plus lifetime US-based engineer support. Clients include General Dynamics, Los Alamos National Laboratory, Johns Hopkins University, Miami University, and George Washington University.
Does VRLA Tech ship VRAM-configured systems nationwide?
Yes. VRLA Tech builds in Los Angeles and ships AI workstations and GPU servers across the United States, including pre-tested configurations at the 24GB, 48GB, 96GB, and datacenter HBM tiers. Every system is burn-in tested for 48 hours before shipment, arrives configured for the customer's exact model and workload, and ships with a 3-year parts warranty plus lifetime US-based engineer support. VRLA Tech has operated since 2016 and counts General Dynamics, Los Alamos, and Johns Hopkins among its clients.
Need help sizing VRAM for your model? VRLA Tech has been building 24GB to 192GB AI systems in Los Angeles since 2016.
Request a VRAM-sized quote →
Request a VRAM-sized quote →
