Skip to content

LLM VRAM Calculator

Will an LLM run on your GPU or Mac, and how many GPUs does it need? Any Hugging Face model, with the real KV cache of today's architectures.

Trending on Hugging Face

Estimates, not guarantees. Real usage depends on your engine and settings: keep some headroom and confirm with nvidia-smi. Confirm with nvidia-smi.

Estimator
Inkling 2 Mini (MoE)
Inferencellama.cpp / Ollama / LM Studio1 × 32 GB

What do you want to know?

Your setup

Your machine, then a model.

Your machine

Auto picks the fewest GPUs that fit: tensor parallel inside a server (up to 8), pipeline parallel across servers.
Used when the model does not fit in GPU memory: llama.cpp then keeps part of it in RAM, slower. The detect script reads it for you.
A desktop and a browser take 0.5 to 1.5 GB on the card that drives the screen. On Windows, an overflow then spills into shared system memory and generation slows down sharply instead of failing; on Linux, loading fails.

Model

Paste a link or org/model: reads the model's config.json and safetensors metadata straight from Hugging Face.

Thinking Machines Lab, United States

26.4B parameters (3.4B active) · 1M context · Apache 2.0

Looped MoE: 24 layers, the middle 12 run 3 times (48 layer visits, each with its own cache) · 128 experts · 8 global + 40 sliding-window visits (1024 tokens), GQA 8 KV × 128 · short convolutions · image input

Context

How much text the model keeps in mind: your prompt, documents and its answer. 32K tokens ≈ 25,000 words.

Using GGUF Q4_K_M weights (picked to fit), 1 request at a time, BF16 / FP16 KV cache, llama.cpp.

Measured on your GPU

Inkling 2 Mini (MoE) has been measured on a GeForce RTX 5090. Across 4 measurements on this card, our estimate is within 20% (median).

Published measurements on this GPU against our estimate
MeasurementRealEstimate
Inkling 2 Mini · generation · llama.cpp · UD-IQ4_XS96.9 tok/s91.1 tok/s
Inkling 2 Mini · prompt · llama.cpp · UD-IQ4_XS2,446 tok/s3,368 tok/s
gpt-oss 20B · generation · llama.cpp · MXFP4 GGUF283 tok/s322 tok/s
gpt-oss 20B · prompt · llama.cpp · MXFP4 GGUF9,848 tok/s12,378 tok/s

Published benchmarks, each with its source in How accurate is it? below.

Your answer

Yes or no, how fast, and what else fits.

On your machine

Fits

Yes, it runs

1 × GeForce RTX 5090 · 32 GB · GGUF Q4_K_M · about 74 tokens/s

Estimate: keep some headroom, and confirm with nvidia-smi.

Or rent it: $0.31/h on Vast.aiOn-demand prices, checked every hour. Prices change: check before you rent.Compare 3 providers →

Speed

Drafts tokens ahead, checks them in one pass. Costs the draft weights plus ~2 GiB per GPU (vLLM logs). Measured gain on InferenceX, 510 runs with and without it: ×1.71 at 1–4 requests, ×1.48 at 8–16, ×1.33 past 64. Less on unpredictable text.
Per request Generation speed seen by one user, at full context≈ 74 tok/s
First token, 32K prompt Time to read a prompt as long as the chosen context before the first token appears (prefill), for one request. It depends on the GPU's compute (TFLOPS), not its memory bandwidth.≈ 11 s

Same model through an API: $0.300 per 1M output tokens on Primeintellect Cheapest of 3 API offers (OpenRouter and Hugging Face Inference Providers): $0.075 per 1M input tokens, $0.3 per 1M output tokens.

Compare 2 API providers: price, speed, first token

Estimate (±30–50%, no speculative decoding): generation at full context from memory bandwidth (1.792 TB/s per GPU) and llama.cpp / Ollama / LM Studio efficiency, first token from llama.cpp's measured speed on this GPU. API prices and speeds: OpenRouter and Hugging Face Inference Providers.

VRAM per GPU

  • Weights15.9 GB
  • KV cache1.23 GB
  • Recurrent state0.00 GB
  • Runtime overhead2.03 GB
Needed19.2 GB
Usable Memory per GPU × usable %32.0 GB
Headroom12.8 GB
Longest context here With these weights, on your GPU memory alone452,264 tokens

Median gap against 141 real launches: 1.3% on weights. How is this computed?

Watch it generate

One answer as a user sees it, after a 1K-token prompt. You read about 5 tokens a second.

1 × GeForce RTX 5090: about 74 tokens per second, first token after 294 ms; Io-net API: about 111 tokens per second, first token after 806 ms.

A 500-token answer takes about 7.0 s here, 5.3 s with Io-net.

Simulated in your browser from the estimate above (the text is only filler). API speed and first token: measured by Hugging Face Inference Providers. Reading speed: 238 words a minute (Brysbaert, 2019).

Measured on your GPU

Inkling 2 Mini (MoE) has been measured on a GeForce RTX 5090. Across 4 measurements on this card, our estimate is within 20% (median).

Published measurements on this GPU against our estimate
MeasurementRealEstimate
Inkling 2 Mini · generation · llama.cpp · UD-IQ4_XS96.9 tok/s91.1 tok/s
Inkling 2 Mini · prompt · llama.cpp · UD-IQ4_XS2,446 tok/s3,368 tok/s
gpt-oss 20B · generation · llama.cpp · MXFP4 GGUF283 tok/s322 tok/s
gpt-oss 20B · prompt · llama.cpp · MXFP4 GGUF9,848 tok/s12,378 tok/s

Published benchmarks, each with its source in How accurate is it? below.

Your options

Formats and context lengths that fit, and what else runs on your machine. Click to apply.

Does it fit? Weights × context on 1 × GeForce RTX 5090

Weights4K8K32K128K256K1M
BF16 / FP16
GGUF Q8_0
GGUF Q6_K
GGUF Q5_K_M
GGUF Q4_K_M
GGUF Q3_K_M

Longest context that fits

  • BF16 / FP16 up to 1M with RAM
  • GGUF Q8_0 up to 128K (tight), 1M with RAM
  • GGUF Q6_K up to 256K (tight), 1M with RAM
  • GGUF Q5_K_M to GGUF Q3_K_M up to 256K, 1M with RAM

✓ fits · ~ tight · ◐ fits with 64 GB of system RAM (slower) · ✕ does not fit. Context per request, 1 concurrent request, llama.cpp / Ollama / LM Studio. Click a cell to apply it. Tap a format to use it at its longest context.

What else runs on your machine

Largest first, each with the most faithful weight format that fits 32K tokens × 1 request.

Quantization on GeForce RTX 5090

Fits on your GPU

With your RAM, slower

Fewer bits per weight: smaller and faster, at some quality cost. The KV cache is set separately.

Run it

The command for Ollama, LM Studio or llama.cpp, and the electricity it uses at home.

Run it at home: electricity

Your electricity price, in dollars per kWh, as on your bill
575 W per card at full load (manufacturer's board power)

Choose your country or enter your price per kWh.

Upper bound: board power at full load; generation usually draws less, the rest of the computer adds some. Household prices, all taxes included, 2,500 to 4,999 kWh a year: Eurostat, first half of 2026. United States: average residential price from January to August 2026, by state (EIA). No figure for the UK: enter yours.

Command

# Ollama: start the server with a 32,768-token context, then run the model in another terminal
OLLAMA_CONTEXT_LENGTH=32768 ollama serve
ollama run hf.co/AtomicChat/Inkling-2-Mini-GGUF:Q4_K_M

New & trending open models

Lab countries from the registry of this site, completed with Epoch AI (opens in a new tab) (CC BY 4.0).

Use it in Claude, Cursor or ChatGPT

Add this calculator to your AI assistant once, then just ask, for example: Can Qwen3.8 27B run on one RTX 4090 with a 32K context? It answers with the same numbers as this page, for the listed models and any Hugging Face model, with GPU rental prices. No key, no sign-up.

Claude (web or desktop): Settings → Connectors → Add custom connector, paste this URL, then turn it on in a chat with the + button.

https://studiotvai.com/api/mcp

The same data as JSON: /api/models.json (every model, with the link to its file), /api/gpus.json, and one file per model such as /api/models/llama-3-3-70b.json. Each model page also has a VRAM badge to paste in a model card.

How accurate is it? Estimates vs real measurements

Every published measurement we can find goes in this table, next to the calculator's estimate for the same model, GPU, engine and file. Median gap: 8% on generation speed (33 measurements) and 10.3% on prompt speed (20). The misses stay in the table: they show where an engine is still catching up with a new architecture. Server launches are checked separately, every day, against 141 runs published by InferenceX: median gap 1.3% on weights per GPU, 1.9% on the KV cache allocated, 13.9% on speed.

SetupMeasuredEstimateGap
llama.cpp guide for gpt-oss (llama-bench) (opens in a new tab) · 2025-08
gpt-oss 20B · RTX PRO 6000 Blackwell · llama.cpp · MXFP4 GGUF287 tok/s305 tok/s+6%
gpt-oss 20B · RTX PRO 6000 Blackwell · llama.cpp · MXFP4 GGUF · 2,048-token prompt11,522 tok/s13,741 tok/s+19%
gpt-oss 120B · RTX PRO 6000 Blackwell · llama.cpp · MXFP4 GGUF196 tok/s222 tok/s+13%
gpt-oss 120B · RTX PRO 6000 Blackwell · llama.cpp · MXFP4 GGUF · 2,048-token prompt5,518 tok/s6,809 tok/s+23%
gpt-oss 20B · GeForce RTX 3090 · llama.cpp · MXFP4 GGUF162 tok/s173 tok/s+7%
gpt-oss 20B · GeForce RTX 3090 · llama.cpp · MXFP4 GGUF · 2,048-token prompt5,171 tok/s4,597 tok/s−11%
gpt-oss 20B · GeForce RTX 4080 / 4080 SUPER · llama.cpp · MXFP4 GGUF187 tok/s148 tok/s−20%
gpt-oss 20B · GeForce RTX 4080 / 4080 SUPER · llama.cpp · MXFP4 GGUF · 2,048-token prompt8,171 tok/s7,612 tok/s−7%
gpt-oss 20B · GeForce RTX 5060 Ti · llama.cpp · MXFP4 GGUF112 tok/s95.7 tok/s−14%
gpt-oss 20B · GeForce RTX 5060 Ti · llama.cpp · MXFP4 GGUF · 2,048-token prompt3,839 tok/s3,469 tok/s−10%
gpt-oss 20B · GeForce RTX 5070 Ti · llama.cpp · MXFP4 GGUF190 tok/s188 tok/s−1%
gpt-oss 20B · GeForce RTX 5070 Ti · llama.cpp · MXFP4 GGUF · 2,048-token prompt6,340 tok/s6,962 tok/s+10%
gpt-oss 20B · GeForce RTX 5080 · llama.cpp · MXFP4 GGUF205 tok/s193 tok/s−6%
gpt-oss 20B · GeForce RTX 5080 · llama.cpp · MXFP4 GGUF · 2,048-token prompt7,477 tok/s7,845 tok/s+5%
gpt-oss 20B · GeForce RTX 5090 · llama.cpp · MXFP4 GGUF283 tok/s322 tok/s+14%
gpt-oss 20B · GeForce RTX 5090 · llama.cpp · MXFP4 GGUF · 2,048-token prompt9,848 tok/s12,378 tok/s+26%
gpt-oss 20B · Mac Studio M3 Ultra 512 GB · llama.cpp · MXFP4 GGUF116 tok/s106 tok/s−9%
gpt-oss 20B · Mac Studio M3 Ultra 512 GB · llama.cpp · MXFP4 GGUF · 2,048-token prompt2,816 tok/s2,568 tok/s−9%
gpt-oss 20B · Mac Studio M2 Ultra 192 GB · llama.cpp · MXFP4 GGUF116 tok/s107 tok/s−8%
gpt-oss 20B · Mac Studio M2 Ultra 192 GB · llama.cpp · MXFP4 GGUF · 2,048-token prompt2,191 tok/s2,161 tok/s−1%
gpt-oss 120B · Mac Studio M2 Ultra 192 GB · llama.cpp · MXFP4 GGUF79.7 tok/s76.6 tok/s−4%
gpt-oss 120B · Mac Studio M2 Ultra 192 GB · llama.cpp · MXFP4 GGUF · 2,048-token prompt1,245 tok/s1,071 tok/s−14%
gpt-oss 20B · Mac M1 Max 64 GB · llama.cpp · MXFP4 GGUF75.2 tok/s66.8 tok/s−11%
gpt-oss 20B · Mac M1 Max 64 GB · llama.cpp · MXFP4 GGUF · 2,048-token prompt995 tok/s925 tok/s−7%
Strix Halo guide (llama-bench, Vulkan) (opens in a new tab) · 2026
Qwen3-Coder 30B-A3B · Ryzen AI Max+ 395 / 392 / 388 128 GB · llama.cpp · Q4_K_M94.6 tok/s67.7 tok/s−28%
Qwen3-Coder-Next 80B-A3B · Ryzen AI Max+ 395 / 392 / 388 128 GB · llama.cpp · Q4_K_M62.1 tok/s59.1 tok/s−5%
Qwen3.8-Flash-Next · Ryzen AI Max+ 395 / 392 / 388 128 GB · llama.cpp · Q4_K_M27.2 tok/s38.5 tok/s+42%
gpt-oss 120B · Ryzen AI Max+ 395 / 392 / 388 128 GB · llama.cpp · MXFP4 GGUF55.6 tok/s55.4 tok/s0%
Qwen3.6 35B-A3B · Ryzen AI Max+ 395 / 392 / 388 128 GB · llama.cpp · Q4_K_M62.6 tok/s71.1 tok/s+14%
Gemma 4 26B-A4B · Ryzen AI Max+ 395 / 392 / 388 128 GB · llama.cpp · Q4_K_M74.0 tok/s54.0 tok/s−27%
Nemotron 3 Super 120B-A12B · Ryzen AI Max+ 395 / 392 / 388 128 GB · llama.cpp · Q4_K_M18.4 tok/s17.1 tok/s−7%
Llama 4 Scout 17B-16E · Ryzen AI Max+ 395 / 392 / 388 128 GB · llama.cpp · Q4_K_M18.3 tok/s15.2 tok/s−17%
Qwen3-Coder 30B-A3B · Ryzen AI Max+ 395 / 392 / 388 128 GB · llama.cpp · Q4_K_M · 512-token prompt1,264 tok/s1,236 tok/s−2%
Qwen3-Coder-Next 80B-A3B · Ryzen AI Max+ 395 / 392 / 388 128 GB · llama.cpp · Q4_K_M · 512-token prompt676 tok/s804 tok/s+19%
Qwen3.8-Flash-Next · Ryzen AI Max+ 395 / 392 / 388 128 GB · llama.cpp · Q4_K_M · 512-token prompt395 tok/s429 tok/s+9%
Gemma 4 31B · Ryzen AI Max+ 395 / 392 / 388 128 GB · llama.cpp · Q4_K_M · 512-token prompt308 tok/s329 tok/s+7%
GPU Benchmarks on LLM Inference (llama-bench) (opens in a new tab) · 2024-05
Llama 3.3 70B · 2 × GeForce RTX 4090 · llama.cpp · Q4_K_M · Llama 3 70B, same shape as 3.319.1 tok/s17.5 tok/s−8%
Llama 3.3 70B · 2 × GeForce RTX 3090 · llama.cpp · Q4_K_M · Llama 3 70B, same shape as 3.316.3 tok/s16.0 tok/s−2%
NInfer-all: llama.cpp on 13 rented cards (Unsloth build) (opens in a new tab) · 2026-11-21
Inkling 2 Mini · RTX PRO 6000 Blackwell · llama.cpp · UD-IQ4_XS97.2 tok/s88.3 tok/s−9%
Inkling 2 Mini · RTX PRO 6000 Blackwell · llama.cpp · UD-IQ4_XS · 2,048-token prompt2,687 tok/s3,739 tok/s+39%
Inkling 2 Mini · GeForce RTX 5090 · llama.cpp · UD-IQ4_XS96.9 tok/s91.1 tok/s−6%
Inkling 2 Mini · GeForce RTX 5090 · llama.cpp · UD-IQ4_XS · 2,048-token prompt2,446 tok/s3,368 tok/s+38%
Inkling 2 Mini · GeForce RTX 4090 · llama.cpp · UD-IQ4_XS75.0 tok/s72.8 tok/s−3%
Inkling 2 Mini · GeForce RTX 4090 · llama.cpp · UD-IQ4_XS · 2,048-token prompt2,665 tok/s3,323 tok/s+25%
Inkling 2 Mini · RTX 6000 Ada · llama.cpp · UD-IQ4_XS72.0 tok/s70.8 tok/s−2%
Inkling 2 Mini · GeForce RTX 3090 · llama.cpp · UD-IQ4_XS59.5 tok/s66.6 tok/s+12%
Inkling 2 Mini · GeForce RTX 3090 · llama.cpp · UD-IQ4_XS · 2,048-token prompt1,402 tok/s1,251 tok/s−11%
Inkling 2 Mini · RTX A6000 · llama.cpp · UD-IQ4_XS55.9 tok/s62.5 tok/s+12%
Inkling 2 Mini · NVIDIA L40S · llama.cpp · UD-IQ4_XS65.2 tok/s67.3 tok/s+3%
Inkling 2 Mini · NVIDIA A100 80GB · llama.cpp · UD-IQ4_XS47.0 tok/s74.2 tok/s+58%
r/LocalLLM user report, vLLM inkling-2-mini image · 2026-11-21
Inkling 2 Mini · RTX PRO 6000 Blackwell · vLLM · NVFP4 / MXFP8 (official) · no speculative decoding132 tok/s139 tok/s+5%
Inkling 2 Mini · RTX PRO 6000 Blackwell · vLLM · NVFP4 / MXFP8 (official) · 2 requests · 2 requests at once121 tok/s111 tok/s−8%
Inkling 2 Mini · RTX PRO 6000 Blackwell · vLLM · NVFP4 / MXFP8 (official) · 256K context · KV cache pool: 731K tokens2.8 requests2.9 requests+3%
RigBase: M5 Pro Mac mini, llama.cpp b12600 (opens in a new tab) · 2026-11-21
Inkling 2 Mini · Mac M5 Pro 64 GB · llama.cpp · IQ4_NL · upstream llama.cpp, slower than Unsloth's build29.9 tok/s33.4 tok/s+12%
Inkling 2 Mini · Mac M5 Pro 64 GB · llama.cpp · IQ4_NL · process memory, 128K cache reserved21.5 GiB21.4 GiB0%

Gap = estimate ÷ measurement − 1, for one request unless noted. Speeds are llama-bench or server figures as published. Send us yours: a model, a GPU, an engine and a number are enough to add a row.

VRAM requirements by model

GPU memory for the weights alone, in GiB. Each model page details the KV cache by context length and how many GPUs of each type it takes.

ModelParametersBF16FP84-bit
Dense
Meta, United StatesLlama 3.1 8B VRAM8.0B15.08.54.6
Meta, United StatesLlama 3 8B VRAM8.0B15.08.54.6
Meta, United StatesLlama 3.2 3B VRAM3.2B6.03.41.9
Nanbeige LLM Lab (BOSS Zhipin), ChinaNanbeige4.3 33B VRAMNew33B61.733.118.9
IBM, United StatesGranite 4.2 30B VRAM29B54.528.016.5
IBM, United StatesGranite 4.2 8B VRAM8.8B16.49.05.0
IBM, United StatesGranite 4.2 3B VRAM3.7B6.83.92.1
OpenBMB, ChinaMiniCPM5 2B VRAM2.5B4.72.81.5
OpenBMB, ChinaMiniCPM5 1B VRAM1.1B2.01.40.6
Mistral AI, FranceMinistral 3 14B VRAM14B26.014.27.7
Mistral AI, FranceMinistral 3 8B VRAM8.9B16.69.34.9
Mistral AI, FranceMinistral 3 3B VRAM3.9B7.24.02.0
Mistral AI, FranceDevstral 2 123B VRAM125B23311970.6
Mistral AI, FranceDevstral Small 2 24B VRAM24B44.723.613.4
Mistral AI, FranceMistral Medium 3.5 128B VRAM128B23812270.6
Mistral AI, FranceMagistral Small 1.2 24B VRAM24B44.723.613.4
Mistral AI, FranceMistral Small 3 24B VRAM24B43.923.213.4
Mistral AI, FranceMistral Large 2411 123B VRAM123B22811569.0
Mistral AI, FrancePixtral Large 124B VRAM124B23111669.0
Mistral AI, FranceCodestral 22B VRAM22B41.421.112.6
Mistral AI, FranceMistral Small 2409 22B VRAM22B41.421.112.6
Mistral AI, FrancePixtral 12B VRAM13B23.613.17.0
NVIDIA, United StatesNemotron 3.5 Parse VRAM0.6B1.20.70.1
LLaVA (UW–Madison), United StatesLLaVA 1.6 Mistral 7B VRAM7.6B14.17.34.1
Mistral AI, FranceMistral Small 3.2 24B VRAM24B44.723.613.4
Alibaba, ChinaQwen3 32B VRAM33B61.032.018.6
Meta, United StatesLlama 3.3 70B VRAM71B13167.739.9
Meta, United StatesLlama 3.1 70B VRAM71B13167.739.9
Meta, United StatesLlama 3.1 405B VRAM406B756382229
Tencent, ChinaYoutu-Parsing-Omni VRAM5.3B9.95.62.8
Mistral AI, FranceMistral 7B v0.3 VRAM7.2B13.57.04.1
Mistral AI, FranceMistral Small 3.1 24B VRAM24B44.723.613.4
Mistral AI, FranceMistral Nemo VRAM12B22.812.77.0
DeepSeek, ChinaDeepSeek R1 Qwen3 8B VRAM8.2B15.38.84.7
DeepSeek, ChinaDeepSeek R1 Distill Qwen 32B VRAM33B61.032.018.6
DeepSeek, ChinaDeepSeek R1 Distill Qwen 14B VRAM15B27.515.28.5
DeepSeek, ChinaDeepSeek R1 Distill Qwen 7B VRAM7.6B14.28.14.4
DeepSeek, ChinaDeepSeek R1 Distill Llama 8B VRAM8.0B15.08.54.6
Alibaba, ChinaQwen3 4B VRAM4.0B7.54.12.3
Alibaba, ChinaQwen3 8B VRAM8.2B15.38.84.7
Alibaba, ChinaQwen2.5 7B VRAM7.6B14.28.14.4
Alibaba, ChinaQwen3 14B VRAM15B27.515.28.5
Alibaba, ChinaQwen2.5 3B VRAM3.1B5.73.21.8
Alibaba, ChinaQwen2.5 32B VRAM33B61.032.018.6
Alibaba, ChinaQwen2.5 14B VRAM15B27.515.28.5
Alibaba, ChinaQwen3-VL 8B VRAM8.8B16.39.34.7
Alibaba, ChinaQwen2.5-VL 7B VRAM8.3B15.48.74.4
Alibaba, ChinaQwen3-VL 4B VRAM4.4B8.34.52.3
Alibaba, ChinaQwen2.5-VL 3B VRAM3.8B7.03.81.8
Alibaba, ChinaQwen2.5-VL 32B VRAM33B62.332.618.6
Alibaba, ChinaQwen2-VL 7B VRAM8.3B15.48.74.4
Microsoft, United Statesphi-4 VRAM15B27.314.68.3
Microsoft, United StatesPhi-3.5-mini VRAM3.8B7.13.72.2
Microsoft, United StatesPhi-4-mini VRAM3.8B7.14.12.3
Microsoft, United StatesPhi-3-mini-128k VRAM3.8B7.13.72.2
Tencent, ChinaWeVisOne-VL 4B VRAM4.8B9.05.22.6
IBM, United StatesGranite 4.1 3B VRAM3.4B6.33.42.0
Microsoft, United StatesMage-VL-Cowork VRAMNew4.7B8.85.12.6
Alibaba, ChinaQwen2 7B VRAM7.6B14.28.14.4
Alibaba, ChinaQwen2.5 72B VRAM73B13570.043.7
Alibaba, ChinaQwen3-VL 32B VRAM33B62.132.518.6
Alibaba, ChinaQwen2.5 14B-1M VRAM15B27.515.28.5
Alibaba, ChinaQwen2.5-VL 72B VRAM73B13770.743.7
Nous Research, United StatesHermes-3-Llama-3.1 8B VRAM8.0B15.08.54.6
IBM, United StatesGranite 3.3 8B VRAM8.2B15.27.84.6
Alibaba (MNN), ChinaMNN-Pocket 0.3B VRAM0.4B0.70.40.3
KAIST MLP Lab (KORMo), South KoreaKORMo-Web 11B VRAMNew11B20.811.46.2
Baidu, ChinaERNIE 4.5 21B-A3B VRAM22B40.720.612.3
Sliding-window attention
Google, United StatesGemma 2 9B VRAM9.2B17.29.55.4
Google, United StatesGemma 3 4B VRAM4.3B8.04.62.3
Google, United StatesGemma 3 12B VRAM12B22.712.36.8
Google, United StatesGemma 3 27B VRAM27B51.126.915.5
Baidu, ChinaUnlimited-OCR VRAM3.3B (0.57B active)6.23.41.8
Google, United StatesGemma 4 12B VRAM12B22.312.16.9
Google, United StatesGemma 4 31B VRAM31B58.230.417.5
Google, United StatesGemma 4 26B-A4B VRAM26B (4B active)48.124.715.4
OpenAI, United Statesgpt-oss 20B VRAM21B (3.6B active)38.920.610.8
OpenAI, United Statesgpt-oss 120B VRAM117B (5.1B active)21811058.3
Meta, United StatesLlama 4 Scout 17B-16E VRAM109B (17B active)20210360.8
Meta, United StatesLlama 4 Maverick 17B-128E VRAM402B (17B active)748376226
Aleph Alpha, GermanyKolibri-1 VRAM78B (3.46B active)14573.344.0
Recursive, United StatesRecursive Flint VRAM40B (2.5B active)74.137.622.4
NVIDIA, United StatesNemotron Labs Dory 1.0 4B VRAM4.0B7.44.32.3
StepFun, ChinaStep 3.5 Flash VRAM199B (11B active)367184111
StepFun, ChinaStep 3.7 Flash VRAM201B (11B active)371186111
poolside, United StatesLaguna S 2.1 VRAM118B (8B active)21911066.2
poolside, United StatesLaguna XS 2.1 VRAM33B (3B active)62.331.518.8
Cohere, CanadaCommand A+ VRAM219B (25B active)407205123
Thinking Machines Lab, United StatesInkling Small VRAM266B (12B active)491247148
Meta, United StatesMuse Glimmer 30B VRAM30B55.530.216.0
Thinking Machines Lab, United StatesInkling 2 Mini VRAMNew26B (3.4B active)49.225.415.8
Country not known yetHumanizer VRAM12B22.313.06.9
JetBrains, CzechiaMellum2.1 12B-A2.5B VRAM12B (2.5B active)22.611.77.4
iFlytek, ChinaSpark-X2.5 4B VRAM4.1B7.74.12.4
Mistral AI, FranceMinistral 8B VRAM8.0B14.98.54.6
Google, United StatesGemma 4 E4B VRAM8.0B14.98.14.4
Google, United StatesGemma 4 E2B VRAM5.1B9.55.12.7
iFlytek, ChinaSpark-X2.6 4B VRAM4.1B7.74.12.4
iFlytek, ChinaSpark-X2.6 1.7B VRAMNew1.7B3.21.81.0
poolside, United StatesLaguna XS.2 VRAM33B (2.8B active)62.331.518.8
Tencent, ChinaQwen3.8 27B-DSpine-b16 VRAMNew2.0B3.82.81.4
Microsoft, United StatesAion-1.0 2.3B VRAMNew2.3B4.32.51.4
Mixture of experts
Tencent, ChinaHy-Memory 30B-A3B VRAM30B (3B active)56.028.517.0
Moonshot AI, ChinaKimi VL A3B Instruct VRAM16B (3B active)30.615.99.7
Mistral AI, FranceMixtral 8x7B VRAM47B (12.9B active)87.043.726.3
Baidu, ChinaUnlimited-OCR 2 VRAMNew3.3B (0.58B active)6.23.41.8
China Telecom, ChinaXing4.0 29B-A4B VRAM31B (4B active)55.028.416.7
Maxwell (InfiniteAI), SingaporeMaxwell V1 Flash VRAM37B (3B active)67.034.320.3
Alibaba, ChinaQwen3 30B-A3B VRAM31B (3B active)56.929.017.2
NVIDIA, United StatesNemotron 3.5 Lightning 30B-A3B VRAM33B (3B active)58.830.122.0
Z.ai, ChinaGLM-4.5-Air VRAM110B (13B active)19910164.7
Alibaba, ChinaQwen3-VL 30B-A3B VRAM31B (3B active)57.929.517.2
Z.ai, ChinaGLM-4.7-Flash VRAM31B (3.6B active)55.828.516.9
Hybrid linear attention
Alibaba, ChinaQwen3.5 4B VRAM4.7B8.54.82.5
OpenBMB, ChinaMiniCPM-V-4.7 1B VRAM1.6B2.91.90.6
KRAFTON, South KoreaRaon-1 mini preview VRAM29B (4.3B active)53.628.016.3
KRAFTON, South KoreaRaon-1 nano VRAMNew3.1B5.83.31.8
Alibaba, ChinaQwen3.5 9B VRAM9.7B17.510.75.2
Alibaba, ChinaQwen3.8 27B VRAM28B51.027.815.4
Alibaba, ChinaQwen3.6 35B-A3B VRAM36B (3B active)65.433.619.6
DiffuSpace, ChinaClover 35B-A3B Preview VRAMNew35B (3B active)64.633.219.6
SenseTime (SenseNova), Hong Kong SAR ChinaSenseNova Loop 9B Preview VRAMNew9.7B17.510.75.2
Alibaba, ChinaQwen3.5 122B-A10B VRAM125B (10B active)22811668.8
Alibaba, ChinaQwen3.8-Flash-Next VRAM180B (6B active)330166105
AutoTrust AI, SingaporeQwen3.8-Flash-Next E384 FP4 V2 VRAMNew149B (6B active)27413886.7
Liquid AI, United StatesLFM3 1.7B VRAMNew1.7B3.21.91.0
Liquid AI, United StatesLFM3-VL 2B VRAMNew1.9B3.62.01.0
Liquid AI, United Statesd1 2B VRAMNew1.9B3.62.01.0
Liquid AI, United StatesLFM2.5 8B-A1B VRAM8.5B (1.5B active)15.88.14.8
Liquid AI, United StatesLFM2 24B-A2B VRAM24B (2.3B active)44.422.313.4
DeepReinforce, United StatesOrnith 1.5 35B-A3B VRAM36B (3B active)65.433.619.6
DeepReinforce, United StatesOrnith 1.5 9B VRAM9.7B17.510.75.2
Cloudflare, United StatesClef VRAM27B51.027.815.4
Cloudflare, United StatesClef Flash VRAM9.4B17.510.75.2
Liquid AI, United Statesd1 3B VRAM3.1B5.83.21.6
AutoTrust AI, SingaporeJEV 27B VL VRAM28B51.027.815.4
LightOn, FranceLightOnOCR-3 4B VRAM4.5B8.54.82.5
Agnes AI, SingaporeAgnes-3.0-Qwen VRAM180B (7.8B active)330166105
Agnes AI, SingaporeAgnes-3.0-Flash VRAM33B60.832.818.4
Alibaba, ChinaQwen3.6 27B VRAM28B51.027.815.4
Alibaba, ChinaQwen3.5 27B VRAM28B51.027.815.4
Alibaba, ChinaQwen3.5 35B-A3B VRAM36B (3B active)65.433.619.6
OpenBMB, ChinaMiniCPM-V-4.7 35B-A3B VRAM35B (3B active)65.633.719.6
Liquid AI, United StatesLFM2.5 24B-A2B VRAM24B (2B active)44.722.613.5
DeepReinforce, United StatesOrnith-1.5 122B VRAM125B (9B active)22811668.8
Shanghai AI Lab, ChinaSage-A1.5 VRAM36B (2.9B active)65.433.619.6
Shanghai AI Lab, ChinaMLExpert VRAM35B (2.9B active)64.633.219.6
Shanghai AI Lab, ChinaAgents-A1 4B VRAM4.5B8.54.82.5
Ant Group, ChinaLing-3.0-flash-Sante VRAM127B (5.1B active)23211770.0
OpenBMB, ChinaMiniCPM-OCR-1.0 VRAMNew1.3B2.41.40.5
Country not known yetQwen3.5 4B-Qwenputer VRAM9.4B17.410.56.8
SenseTime (SenseNova), Hong Kong SAR China6.8-Code2Video VRAMNew27B51.027.815.4
Alibaba, ChinaQwen3-Next 80B-A3B VRAM81B (3B active)14874.844.9
Country not known yetQwen3.5 0.8B-weight-space-programming VRAMNew0.9B1.61.00.5
ByteDance (Axion), ChinaAxion-1.0 35B VRAM35B (2.9B active)65.433.619.6
Maritaca AI, BrazilS5-nano VRAMNew4.7B8.54.82.5
Hybrid Mamba
NVIDIA, United StatesNemotron 3 Nano 30B-A3B VRAM32B (3B active)58.830.122.0
NVIDIA, United StatesNemotron 3 Super 120B-A12B VRAM124B (12B active)22511375.5
NVIDIA, United StatesNemotron 3 Ultra 550B-A55B VRAM561B (55B active)1,023514309
Mistral AI, FranceMamba-Codestral 7B VRAM7.3B13.67.04.1
NVIDIA, United StatesNemotron 3.5 Super VL 120B-A12B VRAM124B (12B active)22611475.6
NVIDIA, United StatesNemotron 3.5 Bolt 74B-A9B VRAM75B (9B active)13970.741.9
NVIDIA, United StatesNemotron 3 Nano 4B VRAM4.0B7.44.52.7
NVIDIA, United StatesNemotron Nano 9B v2 VRAM8.9B16.69.46.2
Large MoE
MiniMax, ChinaMiniMax M3.1 Flash VRAM427B (23B active)795400240
MiniMax, ChinaMiniMax M3 VRAM427B (23B active)795400240
MiniMax, ChinaMiniMax M2.7 VRAM229B (10B active)426214129
DeepSeek, ChinaDeepSeek R1 VRAM684B (37B active)1,250627377
Moonshot AI, ChinaKimi K2.6 VRAM1,027B (32B active)1,913959577
Z.ai, ChinaGLM-5.3 VRAM753B (40.3B active)1,385694418
Shanghai AI Lab, ChinaAtria Dawn VRAMNew753B (40.3B active)1,385694418
Deep Principle, ChinaHydrogen-1.0 VRAMNew753B (40.3B active)1,385694418
Shanghai AI Lab, ChinaAtria Dawn Preview VRAM753B (40.3B active)1,385694418
Tencent, ChinaHy4 Preview VRAM780B (49B active)1,434718433
Tencent, ChinaHy4 Flash VRAMNew220B (11.3B active)409205123
LG AI Research, South KoreaK-EXAONE 2.0 750B-A37B VRAM749B (37B active)1,395699421
Thinking Machines Lab, United StatesInkling VRAM952B (41B active)1,764884533
Thinking Machines Lab, United StatesInkling 2 Small VRAMNew594B (17B active)1,107555334
Z.ai, ChinaGLM-5.3-Flash VRAM321B (18B active)585294176
DeepSeek, ChinaDeepSeek V4 Flash VRAM291B (13B active)530266160
DeepSeek, ChinaDeepSeek V4 Pro VRAM1,599B (49B active)2,9301,467885
DeepSeek, ChinaDeepSeek V4.1 Flash VRAM763B (16B active)1,395699421
Meituan (LongCat), ChinaLongCat-2.0 VRAM1,776B (48B active)3,0521,529922
Meituan (LongCat), ChinaLongCat-2.5 VRAMNew1,780B (48B active)3,0581,531922
Baidu, ChinaERNIE 5.5 Flash Base VRAM286B (14B active)521262157
Xiaomi, ChinaMiMo-V2.6 Pro VRAM1,024B (42B active)1,904954574
Xiaomi, ChinaMiMo-V2.6 Flash VRAM311B (15B active)577290174
Meta, United StatesMuse Spark Open VRAM822B (42B active)1,531769461
Reflection AI, United StatesBeam 1.0 501B-A23B VRAM501B (23.4B active)934468282
Moonshot AI, ChinaKimi K3 VRAM2,780B (104B active)5,1782,5911,563
Mistral AI, FranceMistral Large 4 VRAMNew1,053B (52B active)1,958981590
Mistral AI, FranceMistral Large 3 VRAM676B (41B active)1,259631379
Mistral AI, FranceMistral Small 4 119B VRAM119B (6.5B active)22211267.0
Mistral AI, FranceLeanstral 1.5 119B-A6B VRAM119B (6.5B active)22211267.0
Mistral AI, FranceMixtral 8x22B VRAM141B (39B active)26213179.1
DeepSeek, ChinaDeepSeek V4 Flash Vision Exp VRAM305B (15B active)530266160
Alibaba, ChinaQwen3.8 2.4T-A95B VRAM2,446B (95B active)4,5072,2571,361
DeepReinforce, United StatesOrnith 1.5 397B VRAM403B (18B active)739371223
Shanghai AI Lab, ChinaIntern S2 397B VRAM403B (18B active)739371223
DeepSeek, ChinaDeepSeek V3 VRAM684B (37B active)1,250627377
DeepSeek, ChinaDeepSeek V3.1 VRAM685B (40B active)1,250627378
Moonshot AI, ChinaKimi K2.7 Code VRAM1,027B (33B active)1,913959577
Moonshot AI, ChinaKimi K2.5 VRAM1,027B (33B active)1,913959577
Moonshot AI, ChinaKimi K2 VRAM1,026B (33B active)1,912958577
Z.ai, ChinaGLM-5.2 VRAM753B (42B active)1,385694418
Z.ai, ChinaGLM-5.1 VRAM754B (42B active)1,386695418
Z.ai, ChinaGLM-5 VRAM754B (42B active)1,386695418
MiniMax, ChinaMiniMax M2.5 VRAM229B (10B active)426214129
MiniMax, ChinaMiniMax M2 VRAM229B (11B active)426214129
Xiaomi, ChinaMiMo-V2.5 VRAM311B (15B active)577290174
DeepReinforce, United StatesOrnith-1.0 397B VRAM397B (11B active)739371223
Alibaba, ChinaQwen3 235B-A22B VRAM235B (22B active)438220132
Alibaba, ChinaQwen3.5 397B-A17B VRAM403B (17B active)739371223
Z.ai, ChinaGLM-4.7 VRAM358B (33B active)657330199
Z.ai, ChinaGLM-4.5 VRAM358B (33B active)657330199
Ant Group, ChinaLing-3.1-flash VRAM572B (24B active)1,050526317
StepFun, ChinaStep 5 Preview VRAM604B (33B active)1,119561337
Tencent, ChinaHy3 VRAM299B (20B active)550276166
Moonshot AI, ChinaKimi K3.1 Pro VRAMNew2,780B (106B active)5,1782,5911,563
DeepSeek, ChinaDeepSeek V3.2 VRAM685B (37B active)1,251627378
poolside, United StatesLaguna M.1 VRAM226B (23B active)421211127
Alibaba, ChinaQwen3-VL 235B-A22B VRAM236B (22B active)439221132
Nex AGI (Shanghai Innovation Institute), ChinaNex-N2.6 VRAMNew321B (17B active)585294176
Shanghai AI Lab, ChinaSage-S2.1 VRAMNew753B (40B active)1,385694418
Tencent, ChinaHy3-preview VRAM299B (20B active)550276166
Tencent, ChinaHy4 VRAMNew774B (48B active)1,436720433
Coding models
Alibaba, ChinaQwen3-Coder-Next 80B-A3B VRAM80B (3B active)14874.844.9
Alibaba, ChinaQwen3-Coder 30B-A3B VRAM31B (3.3B active)56.929.017.2
Alibaba, ChinaQwen3-Coder 480B-A35B VRAM480B (35B active)894449270
Alibaba, ChinaQwen2.5-Coder 7B VRAM7.6B14.28.14.4
Alibaba, ChinaQwen2.5-Coder 14B VRAM15B27.515.28.5
Alibaba, ChinaQwen2.5-Coder 32B VRAM33B61.032.018.6
Alibaba, ChinaQwen2.5-Coder 3B VRAM3.1B5.73.21.8
Country not known yetQwen3.5 4B-GLB-3D-Coder VRAMNew4.5B8.54.82.5

LLM VRAM FAQ

How does the calculator compute VRAM?

It adds up what each GPU must hold, for the engine and the GPU you pick, and checks that it fits in the memory the engine allows itself.

  • Weights: parameters × bits ÷ 8, split across the tensor-parallel and pipeline-parallel GPUs. Official checkpoints and published GGUF files count at their exact file size.
  • KV cache: for each group of layers, layers × width × cached tokens × bytes, times the concurrent requests. Width is 2 × KV heads × head dim (GQA), or one compressed latent plus its RoPE part (MLA).
  • Cached tokens: the whole context on full-attention layers, at most the window on sliding-window layers (a modern engine with a paged KV cache keeps only that), the context ÷ the ratio on compressed layers.
  • SGLang: it sizes the sliding-window cache as a fixed share of the full-attention cache (swa_full_tokens_ratio: 0.8 by default, 0.1 for Inkling), so a long request costs full layers + 0.8 × sliding layers per token. On some GPUs it keeps no separate sliding-window cache for a model (Gemma 3 on Hopper, LFM3): every layer then holds the whole context. For models with a recurrent state, its prefix cache keeps states in 0.9× the KV cache memory (5 states per request).
  • vLLM: on hybrid models its prefix cache keeps 2 copies of the recurrent state per request. On sliding-window layers it reserves the window plus the tokens still in flight (2 batches of 2,048 to 16,384 tokens, depending on the GPU's memory), so a long request costs more than the window alone.
  • Engine reserve: what the engine keeps for itself on that GPU: vLLM's 90% default and its overhead measured on real server logs, SGLang's own mem_fraction_static formula, llama.cpp's 1 GiB --fit margin. (KV cache + state) × requests + weights + overhead must stay under memory × usable %.

How accurate is the calculator?

It is checked every day against real server launches published by InferenceX (SemiAnalysis): across 141 launches, the median gap is 1.3% on weights per GPU, 1.9% on the KV cache allocated and 13.9% on speed per request. On home hardware (llama.cpp, Macs, consumer cards), 33 published generation-speed measurements are within 8% of the estimate (median) and 20 prompt-speed measurements within 10.3%: every one is listed with its source in "How accurate is it?" on this page, misses included. Memory follows each architecture exactly; training activations are a ±30% estimate. Keep some headroom and confirm with nvidia-smi.

How much VRAM do I need to run a 70B model?

Take Llama 3.3 70B. Its weights need 131 GiB in BF16 (2 × H100 80 GB), 67.7 GiB in FP8 and about 39.9 GiB in 4-bit (1 × H100, or 2 × RTX 4090). Then add the KV cache of each request: 2.5 GiB at 8K tokens and 10.0 GiB at 32K.

Can I run an LLM on a 24 GB GPU like the RTX 4090?

Yes, models up to about 35B parameters fit in 4-bit with an 8K-token context: Llama 3.1 8B, Llama 3 8B, Llama 3.2 3B, Nanbeige4.3 33B, Granite 4.2 30B, Granite 4.2 8B, Granite 4.2 3B, MiniCPM5 2B, MiniCPM5 1B, Ministral 3 14B, Ministral 3 8B, Ministral 3 3B, Devstral Small 2 24B, Magistral Small 1.2 24B, Mistral Small 3 24B, Codestral 22B, Mistral Small 2409 22B, Pixtral 12B, Nemotron 3.5 Parse, Gemma 2 9B, Gemma 3 4B, Gemma 3 12B, Gemma 3 27B, LLaVA 1.6 Mistral 7B, Mistral Small 3.2 24B, Qwen3 32B, Hy-Memory 30B-A3B, Kimi VL A3B Instruct, Unlimited-OCR, Unlimited-OCR 2, Xing4.0 29B-A4B, Qwen3.5 4B, MiniCPM-V-4.7 1B, Raon-1 mini preview, Raon-1 nano, Qwen3.5 9B, Qwen3.8 27B, Qwen3.6 35B-A3B, Maxwell V1 Flash, Clover 35B-A3B Preview, SenseNova Loop 9B Preview, LFM3 1.7B, LFM3-VL 2B, d1 2B, LFM2.5 8B-A1B, LFM2 24B-A2B, Gemma 4 12B, Gemma 4 31B, Gemma 4 26B-A4B, gpt-oss 20B, Nemotron Labs Dory 1.0 4B, Laguna XS 2.1, Muse Glimmer 30B, Nemotron 3 Nano 30B-A3B, Mamba-Codestral 7B, Inkling 2 Mini, Qwen3-Coder 30B-A3B, Ornith 1.5 35B-A3B, Ornith 1.5 9B, Clef, Clef Flash, d1 3B, Humanizer, JEV 27B VL, Mellum2.1 12B-A2.5B, LightOnOCR-3 4B, Spark-X2.5 4B, Youtu-Parsing-Omni, Agnes-3.0-Flash, Mistral 7B v0.3, Mistral Small 3.1 24B, Mistral Nemo, Ministral 8B, DeepSeek R1 Qwen3 8B, DeepSeek R1 Distill Qwen 32B, DeepSeek R1 Distill Qwen 14B, DeepSeek R1 Distill Qwen 7B, DeepSeek R1 Distill Llama 8B, Qwen3 4B, Qwen3 8B, Qwen2.5 7B, Qwen3 14B, Qwen2.5 3B, Qwen3.6 27B, Qwen3.5 27B, Qwen2.5-Coder 7B, Qwen2.5 32B, Qwen2.5-Coder 14B, Qwen3.5 35B-A3B, Qwen3 30B-A3B, Qwen2.5 14B, Qwen2.5-Coder 32B, Qwen2.5-Coder 3B, Gemma 4 E4B, Gemma 4 E2B, Nemotron 3 Nano 4B, Nemotron 3.5 Lightning 30B-A3B, MiniCPM-V-4.7 35B-A3B, LFM2.5 24B-A2B, Qwen3-VL 8B, Qwen2.5-VL 7B, Qwen3-VL 4B, Qwen2.5-VL 3B, Qwen2.5-VL 32B, Qwen2-VL 7B, phi-4, Phi-3.5-mini, Phi-4-mini, Phi-3-mini-128k, Spark-X2.6 4B, Sage-A1.5, MLExpert, Agents-A1 4B, MiniCPM-OCR-1.0, WeVisOne-VL 4B, Granite 4.1 3B, Qwen3-VL 30B-A3B, Mage-VL-Cowork, Spark-X2.6 1.7B, Qwen3.5 4B-Qwenputer, 6.8-Code2Video, Qwen3.5 4B-GLB-3D-Coder, GLM-4.7-Flash, Laguna XS.2, Qwen2 7B, Qwen3-VL 32B, Qwen2.5 14B-1M, Nemotron Nano 9B v2, Qwen3.5 0.8B-weight-space-programming, Axion-1.0 35B, Hermes-3-Llama-3.1 8B, Qwen3.8 27B-DSpine-b16, Granite 3.3 8B, MNN-Pocket 0.3B, S5-nano, KORMo-Web 11B, ERNIE 4.5 21B-A3B, Aion-1.0 2.3B. Larger models need several GPUs or a card with more memory.

How do I know which GPU and how much VRAM I have?

Click Detect my config in the calculator and paste the one-line command in a terminal: curl -fsSL https://studiotvai.com/detect.sh | sh on macOS and Linux, irm https://studiotvai.com/detect.ps1 | iex in Windows PowerShell. It reads your GPU, its memory, how many you have and your RAM, then opens the calculator set to your machine. Nothing is installed or sent anywhere else. On a Mac, the unified memory is what the GPU uses.

Can I run an LLM that does not fit in my GPU memory?

Often, yes, with llama.cpp (and so Ollama and LM Studio): it keeps part of the model in system RAM and runs the rest on the GPU. It is slower, because each token reads the RAM part over the much slower memory bus. Mixture-of-experts models suffer least: only their active experts are read from RAM. The calculator shows the split, the speed and the llama.cpp command for your RAM, and picks the most faithful format that runs this way.

Ollama, LM Studio or llama.cpp: which one should I use?

All three run the same engine, llama.cpp, with GGUF files, so the memory and speed estimates are the same. Ollama is the simplest from a terminal (one command downloads and runs the model), LM Studio adds a desktop app with a chat window, and llama.cpp itself gives every setting, such as RAM offload flags. The calculator gives the command for each. vLLM and SGLang are for servers: faster with many users, but they need the whole model in GPU memory.

How many GPUs do I need to run an LLM?

Each GPU must hold its share of the weights plus the KV cache of every concurrent request, under the memory the serving engine allows itself (90% by default in vLLM). The calculator tries 1, 2, 4, 8 GPUs and more, splitting the model with tensor parallelism inside a server and pipeline parallelism across servers, and returns the fewest that fit.

What drives LLM VRAM usage?

Four things decide how much memory a model needs once it is serving requests.

  • Weights are fixed: parameters × bits. “4-bit” formats really cost 4.2 to 4.9 bits once scales are counted.
  • KV cache grows with context × concurrent sequences; it decides how many users a GPU can serve.
  • Tensor parallelism splits GQA heads but never below one head per GPU, and replicates MLA latents.
  • Serving engines leave memory free on purpose: vLLM uses 90% of each GPU by default.

Why is VRAM higher than the model size on disk?

The file holds only the weights. At serving time you also pay for the KV cache, which grows with context × concurrent sequences, runtime buffers such as CUDA graphs and activation workspace, and the share of memory the engine leaves free on purpose.

Why are most LLM VRAM formulas wrong?

The classic estimate, 2 × layers × KV heads × head dim × bytes per token, assumes every layer attends to the whole context. Most models released since 2025 break that rule:

  • Hybrid linear attention (Qwen 3.5–3.8, Nemotron 3, Kimi K3): only 1 layer in 4, or fewer, keeps a KV cache; the rest hold a fixed-size state.
  • Sliding windows (Gemma 4, gpt-oss): most layers keep only the last 128 to 1,024 tokens.
  • MLA (GLM-5.3, Kimi K3): one 576-wide compressed latent per token instead of keys and values.
  • Compressed attention (DeepSeek V4): a 128-token window plus the rest of the sequence compressed 4× or 128×.

Why does the KV cache differ so much between models of the same size?

2026 architectures cache very different amounts per token. Qwen 3.5+ keeps a KV cache on only 1 layer in 4, Gemma 4 and gpt-oss limit most layers to a short sliding window, MLA models store one compressed latent instead of keys and values, and DeepSeek V4 compresses the sequence itself. At 128K context, Llama 3.3 70B needs 40 GiB of KV per sequence; Qwen3.8 27B needs 8 GiB.

Can quantization make a model fit on fewer GPUs?

Yes. Weights take parameters × bits per weight, so going from BF16 to 8 bits halves them and 4-bit formats cut them by about 3.5×. As a rule of thumb, 8-bit is near lossless, 5 to 6-bit loses very little and 4-bit loses a little more, depending on the model. Set a fixed number of GPUs and the calculator shows the most accurate format that fits.

Does quantizing the weights shrink the KV cache?

No. The weight format (FP8, NVFP4, GGUF Q4_K_M…) and the KV cache precision are set independently. An FP8 KV cache halves the cache whatever the weights are stored in, at a small accuracy cost.

How much memory does fine-tuning take?

Full fine-tuning with AdamW in mixed precision costs about 16 bytes per parameter before activations: BF16 weights and gradients, an FP32 master copy and two FP32 optimizer moments. LoRA keeps the frozen model at 2 bytes per parameter plus 16 per trainable one, QLoRA at about 0.52. Activations then scale with sequence length × batch, not with parameter count: about layers × 34 × sequence × batch × hidden bytes, or layers × 2 × sequence × batch × hidden plus one layer with gradient checkpointing, plus FP32 logits of sequence × batch × vocabulary × 4 bytes. They are a ±30% estimate: they depend on the MLP shape, fused kernels and the framework, and a fused cross-entropy (Liger) removes most of the logits term.