LLM VRAM Calculator

It adds up what each GPU must hold, for the engine and the GPU you pick, and checks that it fits in the memory the engine allows itself.

• Weights: parameters × bits ÷ 8, split across the tensor-parallel and pipeline-parallel GPUs. Official checkpoints and published GGUF files count at their exact file size.
• KV cache: for each group of layers, layers × width × cached tokens × bytes, times the concurrent requests. Width is 2 × KV heads × head dim (GQA), or one compressed latent plus its RoPE part (MLA).
• Cached tokens: the whole context on full-attention layers, at most the window on sliding-window layers (a modern engine with a paged KV cache keeps only that), the context ÷ the ratio on compressed layers.
• Engine reserve: what the engine keeps for itself on that GPU: vLLM's 90% default and its overhead measured on real server logs, SGLang's own mem_fraction_static formula, llama.cpp's 1 GiB --fit margin. (KV cache + state) × requests + weights + overhead must stay under memory × usable %.

It is checked every day against real server launches published by InferenceX (SemiAnalysis): across 133 launches, the median gap is 1.3% on weights per GPU, 1.8% on the KV cache allocated and 13.2% on speed per request. Memory follows each architecture exactly; speed is an estimate from memory bandwidth (±30–50%), and training activations a ±30% estimate. Keep some headroom and confirm with nvidia-smi.

Take Llama 3.3 70B. Its weights need 131 GiB in BF16 (2 × H100 80 GB), 67.7 GiB in FP8 and about 39.9 GiB in 4-bit (1 × H100, or 2 × RTX 4090). Then add the KV cache of each request: 2.5 GiB at 8K tokens and 10.0 GiB at 32K.

Yes, models up to about 35B parameters fit in 4-bit with an 8K-token context: Llama 3.1 8B, Llama 3 8B, Llama 3.2 3B, Gemma 2 9B, Gemma 3 4B, LLaVA 1.6 Mistral 7B, Mistral Small 3.2 24B, Qwen3 32B, Hy-Memory 30B-A3B, Xing4.0 29B-A4B, Qwen3.5 4B, MiniCPM-V-4.7 1B, Qwen3.5 9B, Qwen3.8 27B, Qwen3.6 35B-A3B, LFM2.5 8B-A1B, LFM2 24B-A2B, Gemma 4 12B, Gemma 4 31B, Gemma 4 26B-A4B, gpt-oss 20B, Nemotron Labs Dory 1.0 4B, Muse Glimmer 30B, Nemotron 3 Nano 30B-A3B, Qwen3-Coder 30B-A3B, Ornith 1.5 35B-A3B, Ornith 1.5 9B, Clef, Clef Flash, D1 3B, Humanizer, JEV 27B VL, Mellum2.1 12B-A2.5B, LightOnOCR-3 4B, Spark-X2.5 4B, Youtu-Parsing-Omni, Agnes-3.0-Flash, Mistral 7B v0.3, Mistral Small 3.1 24B, Mistral Nemo, Ministral 8B, DeepSeek R1 Qwen3 8B, DeepSeek R1 Distill Qwen 32B, DeepSeek R1 Distill Qwen 14B, DeepSeek R1 Distill Qwen 7B, DeepSeek R1 Distill Llama 8B, Qwen3 4B, Qwen3 8B, Qwen2.5 7B, Qwen3 14B, Qwen2.5 3B, Qwen3.6 27B, Qwen3.5 27B, Qwen2.5-Coder 7B, Qwen2.5 32B, Qwen2.5-Coder 14B, Qwen3.5 35B-A3B, Qwen3 30B-A3B, Qwen2.5 14B, Qwen2.5-Coder 32B, Qwen2.5-Coder 3B, Gemma 4 E4B, Gemma 4 E2B, Nemotron 3 Nano 4B, Nemotron 3.5 Lightning 30B-A3B, MiniCPM-V-4.7 35B-A3B, LFM2.5 24B-A2B, Qwen3-VL 8B, Qwen2.5-VL 7B, Qwen3-VL 4B, Qwen2.5-VL 3B, Qwen2.5-VL 32B, Qwen2-VL 7B, phi-4, Phi-3.5-mini, Phi-4-mini, Phi-3-mini-128k, Unlimited-OCR, Spark-X2.6 4B. Larger models need several GPUs or a card with more memory.

Each GPU must hold its share of the weights plus the KV cache of every concurrent request, under the memory the serving engine allows itself (90% by default in vLLM). The calculator tries 1, 2, 4, 8 GPUs and more, splitting the model with tensor parallelism inside a server and pipeline parallelism across servers, and returns the fewest that fit.

Four things decide how much memory a model needs once it is serving requests.

• Weights are fixed: parameters × bits. “4-bit” formats really cost 4.2 to 4.9 bits once scales are counted.
• KV cache grows with context × concurrent sequences; it decides how many users a GPU can serve.
• Tensor parallelism splits GQA heads but never below one head per GPU, and replicates MLA latents.
• Serving engines leave memory free on purpose: vLLM uses 90% of each GPU by default.

The file holds only the weights. At serving time you also pay for the KV cache, which grows with context × concurrent sequences, runtime buffers such as CUDA graphs and activation workspace, and the share of memory the engine leaves free on purpose.

The classic estimate, 2 × layers × KV heads × head dim × bytes per token, assumes every layer attends to the whole context. Most models released since 2025 break that rule:

• Hybrid linear attention (Qwen 3.5–3.8, Nemotron 3, Kimi K3): only 1 layer in 4, or fewer, keeps a KV cache; the rest hold a fixed-size state.
• Sliding windows (Gemma 4, gpt-oss): most layers keep only the last 128 to 1,024 tokens.
• MLA (GLM-5.3, Kimi K3): one 576-wide compressed latent per token instead of keys and values.
• Compressed attention (DeepSeek V4): a 128-token window plus the rest of the sequence compressed 4× or 128×.

2026 architectures cache very different amounts per token. Qwen 3.5+ keeps a KV cache on only 1 layer in 4, Gemma 4 and gpt-oss limit most layers to a short sliding window, MLA models store one compressed latent instead of keys and values, and DeepSeek V4 compresses the sequence itself. At 128K context, Llama 3.3 70B needs 40 GiB of KV per sequence; Qwen3.8 27B needs 8 GiB.

Yes. Weights take parameters × bits per weight, so going from BF16 to 8 bits halves them and 4-bit formats cut them by about 3.5×. As a rule of thumb, 8-bit is near lossless, 5 to 6-bit loses very little and 4-bit loses a little more, depending on the model. Set a fixed number of GPUs and the calculator shows the most accurate format that fits.

No. The weight format (FP8, NVFP4, GGUF Q4_K_M…) and the KV cache precision are set independently. An FP8 KV cache halves the cache whatever the weights are stored in, at a small accuracy cost.

Full fine-tuning with AdamW in mixed precision costs about 16 bytes per parameter before activations: BF16 weights and gradients, an FP32 master copy and two FP32 optimizer moments. LoRA keeps the frozen model at 2 bytes per parameter plus 16 per trainable one, QLoRA at about 0.52. Activations then scale with sequence length × batch, not with parameter count: about layers × 34 × sequence × batch × hidden bytes, or layers × 2 × sequence × batch × hidden plus one layer with gradient checkpointing, plus FP32 logits of sequence × batch × vocabulary × 4 bytes. They are a ±30% estimate: they depend on the MLP shape, fused kernels and the framework, and a fused cross-entropy (Liger) removes most of the logits term.