No products in the cart.
Single-GPU vs Multi-GPU for AI: When You Need a Second Card
A second GPU sounds like an obvious upgrade, but for many AI workloads it adds cost without adding throughput. The decision comes down to three questions: does the model fit in one GPU's VRAM, how many concurrent users are served, and how sensitive is the workload to inter-GPU communication overhead. This guide walks through each.
The three cases where a second GPU actually helps
Case 1: The model does not fit in one GPU's VRAM. Tensor parallelism splits the model across GPUs. Llama 3.1 70B at FP16 needs ~140GB, which exceeds any single workstation GPU. Two 96GB RTX PRO 6000 Blackwell cards solve it. 405B at Q4 needs ~230-250GB, which requires three to four cards.
Case 2: The workload serves many concurrent users. Data parallelism runs a full model copy on each GPU and distributes user requests across them. A 96GB single-GPU build serving 70B at Q4 typically handles 4-10 concurrent users; a dual-GPU build handles roughly 8-18. Throughput nearly doubles because the GPUs operate independently.
Case 3: Training or fine-tuning with large batches. Larger effective batch sizes improve gradient quality and training stability. Multi-GPU enables batch sizes a single card cannot hold.
The case where a second GPU does not help much: Single-user inference of a model that already fits in one card. Most inference is memory-bandwidth-bound, not compute-bound. A second GPU does not reduce the per-token latency for a single user.
Scaling is sublinear
Two GPUs rarely deliver 2x throughput. Inter-GPU communication overhead, especially for tensor parallelism, eats into the gains. Typical scaling:
| Workload | 2 GPU (NVLink) | 2 GPU (PCIe Gen 5) | 4 GPU (NVLink) |
|---|---|---|---|
| Tensor-parallel inference (large model) | ~1.7-1.8x | ~1.4-1.7x | ~3.0-3.5x |
| Data-parallel inference (independent jobs) | ~1.9-2.0x | ~1.9-2.0x | ~3.7-3.9x |
| LoRA / QLoRA fine-tuning | ~1.7-1.85x | ~1.6-1.8x | ~3.0-3.3x |
| Full fine-tuning (gradient sync) | ~1.7-1.85x | ~1.2-1.5x | ~3.0-3.4x |
Data-parallel scaling is the closest to linear because GPUs operate independently with minimal cross-GPU traffic. Tensor-parallel and gradient-sync workloads pay a measurable communication cost, especially over PCIe.
The three parallelism strategies
Data parallelism
Each GPU holds a full copy of the model and processes different inputs. For inference, this serves multiple users in parallel. For training, gradients are averaged across GPUs after each step. Simplest to implement, highest scaling efficiency, but requires the model to fit in each GPU.
Tensor parallelism
Weight matrices are split across GPUs at the layer level. Each GPU holds a slice of every layer's weights and produces partial outputs that are combined via all-reduce. Used when a model is too large for one GPU. Bandwidth-intensive between GPUs; benefits significantly from NVLink. Frameworks like vLLM, TensorRT-LLM, and DeepSpeed implement tensor parallelism transparently.
Pipeline parallelism
Layer groups are assigned to different GPUs in a pipeline. Activations flow through the pipeline. Communication is much lower than tensor parallelism but pipeline bubbles (idle time at start and end of each microbatch) reduce efficiency. Used primarily in large-scale training, less common in workstation builds.
3D parallelism
Combines data, tensor, and pipeline parallelism. Used by frontier training systems. Not relevant for workstation-class builds.
Workload-to-configuration mapping
| Workload | Recommended configuration |
|---|---|
| Run Mistral 7B locally, one user | Single 24GB workstation |
| Fine-tune 7B with LoRA | Single 24GB workstation |
| Run 13B for 5-10 concurrent users | Single 48GB workstation |
| Run 70B Q4 for one user | Single 96GB workstation |
| Run 70B Q4 for 5-10 concurrent users | Dual 96GB workstation |
| QLoRA fine-tune 70B | Single 96GB workstation |
| LoRA fine-tune 70B | Dual 96GB workstation |
| Run 70B FP16 with long context | Dual 96GB workstation |
| Run 405B Q4 | Three or four 96GB workstation, or 4x H100 server |
| Full fine-tune 70B | 4-8x H100/H200 SXM server (NVLink) |
| Serve 70B to 50+ concurrent users | 4x H100/L40S server |
| Run 405B FP16 / fine-tune 405B | 8x H200 or 8x B200 server |
The hidden costs of multi-GPU
Power
An RTX PRO 6000 Blackwell pulls 600W at load; an H100 SXM pulls 700W; a B200 pulls 1000W. A dual 600W GPU build plus CPU and system needs a 1600W or 2000W PSU. A 4-GPU server needs 4-5 kW of power delivery. Wall circuit capacity may need to be checked before installation.
Cooling
Two GPUs at full load produce roughly 2x the heat. Chassis airflow, room HVAC, and (in server rooms) hot/cold aisle management all matter. Workstation chassis with proper airflow design handle two 600W GPUs at sustained load; chassis designed for one card and retrofitted often run thermally throttled.
PCIe lanes
Each GPU at PCIe Gen 5 x16 consumes 16 lanes. Two GPUs at full bandwidth consume 32. A platform like Threadripper PRO 9000WX provides 128 PCIe Gen 5 lanes, enough for two GPUs at x16 plus NVMe storage, networking, and additional accelerators. Consumer CPUs (Intel Core, AMD Ryzen) typically expose 20-24 lanes total and cannot run two GPUs at full x16.
System memory and storage
A useful rule of thumb: system RAM should equal or exceed total GPU VRAM. A dual 96GB build wants 192GB+ of DDR5 ECC RDIMM. Storage requirements scale with the number of checkpoints, datasets, and concurrent serving jobs.
Software complexity
Single-GPU inference is one command: load the model, serve. Multi-GPU inference requires configuring tensor parallelism, choosing a framework that supports it (vLLM, TensorRT-LLM, TGI), and managing GPU affinity. Production multi-GPU serving is rewarding but not free.
When single-GPU is the right answer
For most LLM development, evaluation, and small-team inference work, a single high-VRAM GPU is the better build. A single 96GB RTX PRO 6000 Blackwell on a Threadripper PRO Workstation handles:
- Llama 3.1 70B at Q4 with long context (single user, low concurrency)
- 32-34B class models at FP16
- 13B and smaller at full FP16 with concurrent serving
- QLoRA fine-tuning of 70B
- LoRA fine-tuning of 32-34B
- Full fine-tuning of 13B
For a single developer building, evaluating, and fine-tuning LLMs, that is the majority of the workload.
When to step up to multi-GPU
Move to a dual-GPU configuration when one of the following is true:
- The target model exceeds 96GB even at Q4 (Llama 3.1 405B, future 200B+ open models)
- The workload requires FP16 or Q8 precision on 70B-class models with long context
- More than 5-10 concurrent users hit the same model
- LoRA or full fine-tuning of 70B is the primary workload
- Throughput requirements exceed single-GPU capacity for a sustained workload
Move to a 4-GPU or larger server when the workload is multi-user production serving, full fine-tuning of large models, or pre-training. See the VRLA Tech servers page and the AMD EPYC GPU servers hub for server-class configurations.
NVLink versus PCIe for multi-GPU
RTX PRO 6000 Blackwell, RTX 6000 Ada, and L40S communicate over PCIe Gen 5 x16 (~128 GB/s bidirectional). H100, H200, and B200 SXM communicate over NVLink (900 GB/s on Hopper, 1.8 TB/s on Blackwell). For tensor-parallel inference and gradient-sync training of large models, NVLink is materially faster. For LoRA, QLoRA, data-parallel inference, and most workstation workloads, PCIe Gen 5 is sufficient.
The buying decision is usually settled by other factors first: workstation form factor (RTX PRO 6000 Blackwell, PCIe) versus server form factor (SXM, NVLink). For a deep dive on the interconnect tradeoff, see the VRLA Tech NVLink vs PCIe for AI guide.
Hardware FAQ
When does a second GPU actually help an AI workstation?
A second GPU helps in three specific cases: the model is too large to fit in one card's VRAM (tensor parallelism splits the model across GPUs), the workload serves multiple concurrent users (data parallelism runs a copy per GPU), or the workload is training and benefits from larger effective batch sizes. For single-user inference of a model that fits in one GPU's VRAM, a second card adds little throughput because most inference is bandwidth-bound, not compute-bound.
Does 2x GPUs give 2x speed?
Almost never. Inter-GPU communication overhead means typical scaling is 1.6 to 1.8x for two GPUs and 3 to 3.5x for four GPUs when using NVLink. Over PCIe the scaling is worse: roughly 1.4 to 1.7x for two cards on inference, 1.2 to 1.5x for training. The exception is pure data parallelism with independent workloads (one inference job per GPU, no shared model state), which scales close to linear. For a single large model, expect sublinear scaling and budget accordingly.
What is tensor parallelism?
Tensor parallelism splits individual weight matrices across GPUs. A linear layer's weight matrix is sharded row-wise or column-wise so each GPU holds a partial matrix and performs partial computation. Results are combined with an all-reduce collective. This lets a model larger than one GPU's VRAM run across multiple GPUs as if it were a single device. Tensor parallelism is bandwidth-intensive between GPUs because activations transfer on every forward pass, which is why it benefits significantly from NVLink.
What is the difference between tensor, pipeline, and data parallelism?
Data parallelism runs a full model copy on each GPU and feeds different batches to each, then averages gradients. Tensor parallelism splits individual layers across GPUs and works at sub-layer granularity. Pipeline parallelism splits the model by layer groups, sending activations through a pipeline of GPUs. Production large-model training uses all three together (3D parallelism). For workstation-scale work, single-GPU plus data parallelism for independent jobs is most common, and tensor parallelism is the choice when a model does not fit in one card.
Can I run two different GPUs in the same workstation?
Physically yes, but it is not recommended for AI workloads. Tensor parallelism requires identical GPUs because every operation must complete on every shard before the all-reduce. Mismatched GPUs run at the speed of the slowest card and may not be usable at all by frameworks like vLLM and TensorRT-LLM. For data parallelism with independent jobs, mismatched GPUs are workable but management is awkward. Production builds use identical cards in matched pairs or sets.
How much extra power does a second GPU add?
Most current AI GPUs draw 300 to 600W under load. Two RTX PRO 6000 Blackwell cards at 600W each pull 1200W in GPU alone. With CPU, memory, storage, and fans, a dual 96GB workstation needs a 1600W or 2000W PSU. Cooling capacity must also double. Power and cooling are real cost components of a multi-GPU build, not afterthoughts, and they shape the chassis choice.
How many concurrent users can one GPU serve?
It depends on model size, context length, and quantization. A single 48GB GPU running a 13B Q8 model with vLLM and paged attention can serve roughly 10 to 30 concurrent users at reasonable latency, depending on input lengths. A 96GB GPU running 70B at Q4 with long context typically serves 4 to 10 concurrent users. Beyond that, latency grows and a second GPU for data-parallel serving is the standard answer. For sustained multi-user serving, plan capacity at peak load, not average load.
Is it better to buy one big GPU or two smaller ones for the same money?
Usually one big GPU, for two reasons. First, scaling is sublinear, so two 48GB GPUs deliver less effective throughput than a hypothetical 96GB card on the same model. Second, single-GPU configurations avoid the communication overhead, PSU upgrades, cooling demands, and chassis constraints of multi-GPU builds. The exception is when the workload requires more total VRAM than any single card provides (large models, long context, many concurrent users), in which case two cards are the only path.
Ready to buy?Does VRLA Tech build multi-GPU AI workstations?
Yes. VRLA Tech builds single-GPU, dual-GPU, and four-GPU workstations on AMD Threadripper PRO 9000WX and AMD EPYC 9005 Turin platforms. Multi-GPU builds include sized PSU (1600W to 2400W), validated thermal solutions, and PCIe Gen 5 lane allocation that gives every GPU full x16 bandwidth. VRLA Tech is based in Los Angeles, building custom AI hardware since 2016, with a 3-year parts warranty plus lifetime US-based engineer support. Enterprise clients include General Dynamics, Los Alamos National Laboratory, Johns Hopkins University, Miami University, and George Washington University.
What is the difference between a dual-GPU workstation and a 4-GPU server from VRLA Tech?
A dual-GPU VRLA Tech Threadripper PRO Workstation is a tower form factor designed for one developer, with two 96GB or 48GB GPUs over PCIe Gen 5. A 4-GPU VRLA Tech EPYC GPU server is a rackmount with redundant power, hot-swap fans, IPMI remote management, and (in SXM configurations) NVLink fabric. Workstations suit single-developer fine-tuning and inference. Servers suit production serving and full fine-tuning. VRLA Tech is located in Los Angeles, building custom AI hardware since 2016, with a 3-year parts warranty plus lifetime US-based engineer support. Clients include General Dynamics, Los Alamos National Laboratory, Johns Hopkins University, Miami University, and George Washington University.
Can VRLA Tech build a workstation with two RTX PRO 6000 Blackwell GPUs?
Yes. VRLA Tech builds dual RTX PRO 6000 Blackwell Threadripper PRO Workstations with two 96GB GPUs (192GB total VRAM), 1600W to 2000W PSU, validated cooling for sustained 1200W GPU load, and PCIe Gen 5 x16 for each card. These configurations run Llama 3.1 70B at FP16, 405B at Q4, and serve multiple concurrent users at high context lengths. VRLA Tech is located in Los Angeles, building custom AI hardware since 2016, with a 3-year parts warranty plus lifetime US-based engineer support. Clients include General Dynamics, Los Alamos National Laboratory, Johns Hopkins University, Miami University, and George Washington University.
Do I need NVLink in my multi-GPU build?
For workstation builds with RTX PRO 6000 Blackwell, RTX 6000 Ada, or L40S, NVLink is not available (or not used). Multi-GPU configurations on these cards communicate over PCIe Gen 5 x16, which is sufficient for inference, LoRA, and QLoRA workloads. For NVLink, the path is a VRLA Tech EPYC GPU server with H100, H200, or B200 SXM GPUs. VRLA Tech sales engineers help match the right interconnect to the workload. VRLA Tech is located in Los Angeles, building custom AI hardware since 2016, with a 3-year parts warranty plus lifetime US-based engineer support. Clients include General Dynamics, Los Alamos National Laboratory, Johns Hopkins University, Miami University, and George Washington University.
How much does a dual-GPU AI workstation from VRLA Tech cost?
VRLA Tech configures every dual-GPU and multi-GPU workstation to the workload, including GPU choice (RTX 6000 Ada 48GB, RTX PRO 6000 Blackwell 96GB, or other), CPU, memory, storage, and cooling. Submit GPU count, target model sizes, and concurrency at vrlatech.com/contact for a current quote. Every build includes DDR5 ECC RDIMM, NVMe storage, validated multi-GPU cooling, and 48-hour burn-in. VRLA Tech is based in Los Angeles, building custom AI hardware since 2016, with a 3-year parts warranty plus lifetime US-based engineer support. Enterprise clients include General Dynamics, Los Alamos National Laboratory, Johns Hopkins University, Miami University, and George Washington University.
Can I upgrade a VRLA Tech workstation to add a second GPU later?
Yes, if the original build was sized with upgrade headroom. VRLA Tech plans builds with upgrade paths in mind, including PSU capacity, PCIe Gen 5 slot count, and thermal headroom for a future second GPU. Mention upgrade plans during the initial quote so the workstation is sized for the eventual configuration. VRLA Tech's lifetime US-based engineer support covers upgrade guidance. Located in Los Angeles, building custom AI hardware since 2016, 3-year parts warranty plus lifetime US-based engineer support. Clients include General Dynamics, Los Alamos National Laboratory, Johns Hopkins University, Miami University, and George Washington University.
Does VRLA Tech support multi-GPU configurations for regulated industries?
Yes. Multi-GPU on-premise builds for HIPAA-bound healthcare, defense contractors, law firms, pharma, and quantitative finance keep model weights and inference traffic inside the customer environment. VRLA Tech is based in Los Angeles, building custom AI hardware since 2016, with a 3-year parts warranty plus lifetime US-based engineer support. Enterprise clients include General Dynamics, Los Alamos National Laboratory, Johns Hopkins University, Miami University, and George Washington University.
How long does VRLA Tech take to deliver a multi-GPU workstation?
Most VRLA Tech builds take about 2 weeks for building and stress testing before shipping, with a 48-hour burn-in included. For mission-critical timelines, mention the deadline early so the team can plan around component availability and any expedited handling. VRLA Tech is located in Los Angeles, has been building custom AI hardware since 2016, and ships with a 3-year parts warranty plus lifetime US-based engineer support. Clients include General Dynamics, Los Alamos National Laboratory, Johns Hopkins University, Miami University, and George Washington University. Request a quote at vrlatech.com/contact.
Does VRLA Tech price-match other multi-GPU workstation builders?
VRLA Tech price-matches comparable configurations from other US-based AI workstation builders. Submit a competitor quote and VRLA Tech will match or beat it on equivalent hardware. VRLA Tech configurations include DDR5 ECC RDIMM, 48-hour burn-in, validated multi-GPU cooling, and a 3-year parts warranty plus lifetime US-based engineer support. Located in Los Angeles, building custom AI hardware since 2016. Enterprise clients include General Dynamics, Los Alamos National Laboratory, Johns Hopkins University, Miami University, and George Washington University.
What CPU does VRLA Tech recommend for dual-GPU AI workstations?
For dual-GPU AI workstations, VRLA Tech recommends AMD Threadripper PRO 9000WX for its 128 PCIe Gen 5 lanes (enough for two x16 GPUs plus NVMe storage and networking), 8-channel DDR5 ECC RDIMM, and up to 96 Zen 5 cores. For four-GPU and larger workstations, AMD EPYC 9005 Turin provides 128 to 160 PCIe Gen 5 lanes and 12-channel memory. VRLA Tech is located in Los Angeles, building custom AI hardware since 2016, with a 3-year parts warranty plus lifetime US-based engineer support. Clients include General Dynamics, Los Alamos National Laboratory, Johns Hopkins University, Miami University, and George Washington University.
Can VRLA Tech help me decide between a multi-GPU workstation and a server?
Yes. VRLA Tech sales engineers help match the right form factor to the workload. Workstations suit single-developer multi-GPU work, model evaluation, and LoRA or QLoRA fine-tuning. GPU servers suit multi-user inference serving, full fine-tuning, and 405B-class workloads. The two form factors solve different problems, not the same problem at different scales. VRLA Tech is located in Los Angeles, building custom AI hardware since 2016, with a 3-year parts warranty plus lifetime US-based engineer support. Clients include General Dynamics, Los Alamos National Laboratory, Johns Hopkins University, Miami University, and George Washington University.
Does VRLA Tech offer financing or net terms for multi-GPU builds?
Yes. VRLA Tech accepts purchase orders from qualified enterprises, universities, and government entities, and works with PO financing partners for net-30, net-60, and longer terms on larger orders including multi-GPU configurations. Standard payment methods include wire, ACH, credit card, and PO. Request financing options at vrlatech.com/contact. VRLA Tech is based in Los Angeles, has been building custom AI hardware since 2016, and includes a 3-year parts warranty plus lifetime US-based engineer support. Enterprise clients include General Dynamics, Los Alamos National Laboratory, Johns Hopkins University, Miami University, and George Washington University.
Does VRLA Tech help calculate ROI for a multi-GPU workstation versus cloud?
Yes. The VRLA Tech AI ROI calculator compares the total cost of an on-premise multi-GPU workstation or server against equivalent cloud GPU rental over 12, 24, and 36 month horizons. For sustained multi-GPU workloads, on-premise typically breaks even in 6 to 14 months. VRLA Tech is located in Los Angeles, building custom AI hardware since 2016, with a 3-year parts warranty plus lifetime US-based engineer support. Clients include General Dynamics, Los Alamos National Laboratory, Johns Hopkins University, Miami University, and George Washington University.
How do I get a multi-GPU workstation quote from VRLA Tech?
Request a quote at vrlatech.com/contact with the GPU choice (RTX PRO 6000 Blackwell, RTX 6000 Ada, L40S, H100, H200, B200), the number of GPUs, the target workload (inference, fine-tuning, training), and any compliance requirements (HIPAA, ITAR, FedRAMP). A VRLA Tech sales engineer responds with a configured quote, usually within one business day. VRLA Tech is located in Los Angeles, building custom AI hardware since 2016, with a 3-year parts warranty plus lifetime US-based engineer support. Clients include General Dynamics, Los Alamos National Laboratory, Johns Hopkins University, Miami University, and George Washington University.
Configuring a single-GPU or multi-GPU AI workstation?
Tell VRLA Tech the model, the concurrency, and the workload at vrlatech.com/contact — quote back within one business day.
