Llama 3.1 70B VRAM Requirements

Llama 3.1 70B at a glance

Made by

Meta, United States

Released

July 16, 2024

Parameters

70.5B

Context window

128K tokens

License

Llama 3.1 Community License

Access

Gated: accept the license on Hugging Face (opens in a new tab) before downloading the weights

Reads

Text only

Tool calling

Supported

Languages

English, German, French, Italian, Portuguese, Hindi and 2 more

Knowledge cutoff

December 2023

OpenRouter

Training

7.9 × 10²⁴ FLOP · 15 trillion tokens

Epoch AI (opens in a new tab), CC BY 4.0

On Hugging Face

1.0K likes, 93K downloads last month (opens in a new tab)

Runs with

vLLM, SGLang, llama.cpp (Ollama, LM Studio), MLX (Mac)

Checked against

KV cache matches the GGUF header of bartowski/Meta-Llama-3.1-70B-Instruct-GGUF (opens in a new tab) (what llama.cpp allocates), November 6, 2026

On Ollama

ollama run llama3.1:70b · Q4_K_M, 39.6 GiB 121M pulls (opens in a new tab)

VRAM by precision and context length

Weights plus the KV cache of one request (BF16 cache), in GiB. Add 1 to 3 GiB for the inference engine itself.

Llama 3.1 70B GGUF files

Exact size of each quantization in bartowski/Meta-Llama-3.1-70B-Instruct-GGUF (opens in a new tab), and the smallest GPU that runs it with llama.cpp, Ollama or LM Studio and an 8K-token context.

Read every day from Hugging Face. Split files are added together; vision projectors are left out.

How many GPUs to run Llama 3.1 70B

Fewest GPUs that fit the weights plus one 8K-token request. Data-center GPUs use vLLM (90% of memory usable), the others llama.cpp. Several GPUs split the model with tensor or pipeline parallelism.

8-bit: FP8 with vLLM (INT8 on GPUs without FP8), Q8_0 with llama.cpp. 4-bit: AWQ / GPTQ with vLLM, Q4_K_M with llama.cpp, both counted at the Q4_K_M size.

Cheapest way to run Llama 3.1 70B today

For each GPU: the fewest cards that fit Llama 3.1 70B with one 8K-token request, the most faithful weight format at that count, and the cheapest on-demand price (2026-11-06).

On-demand prices from AWS, Google Cloud, Azure, RunPod, Vast.ai, Oracle Cloud, OVHcloud, Scaleway, Verda, Akash and UpCloud, checked every hour. Serving many users needs more KV cache, so more memory: size it in the calculator. Some links are affiliate links: StudioTV may earn a commission, at no extra cost to you.

AWS, Google Cloud and UpCloud prices: Spare Cores (opens in a new tab), CC BY-SA 4.0 (opens in a new tab), shared under the same license.

Recommended settings for Llama 3.1 70B

Sampling defaults from the official generation_config.json (opens in a new tab) of meta-llama/Llama-3.1-70B-Instruct.

vLLM and SGLang apply these defaults unless a request sets its own. With llama.cpp, pass them yourself: --temp 0.6 --top-p 0.9

KV cache: how memory grows with context

Architecture: 80 layers · GQA, 8 KV heads × 128. Every layer keeps keys and values for every token (grouped-query attention), so the cache grows linearly with context.

Llama 3.1 70B VRAM badge

For a model card or a README: the 4-bit size and the smallest GPU it fits on, linked to this page. Data as JSON: /api/models/llama-3-1-70b.json.

[![Llama 3.1 70B VRAM](https://studiotvai.com/badge/llama-3-1-70b.svg)](https://studiotvai.com/vram-requirements/llama-3-1-70b)

Other models

Same family (dense): Llama 3.1 8B · Llama 3 8B · Llama 3.2 3B · Nanbeige4.3 33B · Granite 4.2 30B · Granite 4.2 8B · Granite 4.2 3B · MiniCPM5 2B · MiniCPM5 1B · Ministral 3 14B · Ministral 3 8B · Ministral 3 3B · Devstral 2 123B · Devstral Small 2 24B · Mistral Medium 3.5 128B · Magistral Small 1.2 24B · Mistral Small 3 24B · Mistral Large 2411 123B · Pixtral Large 124B · Codestral 22B · Mistral Small 2409 22B · Pixtral 12B · Nemotron 3.5 Parse · LLaVA 1.6 Mistral 7B · Mistral Small 3.2 24B · Qwen3 32B · Llama 3.3 70B · Llama 3.1 405B · Youtu-Parsing-Omni · Mistral 7B v0.3 · Mistral Small 3.1 24B · Mistral Nemo · DeepSeek R1 Qwen3 8B · DeepSeek R1 Distill Qwen 32B · DeepSeek R1 Distill Qwen 14B · DeepSeek R1 Distill Qwen 7B · DeepSeek R1 Distill Llama 8B · Qwen3 4B · Qwen3 8B · Qwen2.5 7B · Qwen3 14B · Qwen2.5 3B · Qwen2.5 32B · Qwen2.5 14B · Qwen3-VL 8B · Qwen2.5-VL 7B · Qwen3-VL 4B · Qwen2.5-VL 3B · Qwen2.5-VL 32B · Qwen2-VL 7B · phi-4 · Phi-3.5-mini · Phi-4-mini · Phi-3-mini-128k · WeVisOne-VL 4B · Granite 4.1 3B · Mage-VL-Cowork · Qwen2 7B · Qwen2.5 72B · Qwen3-VL 32B · Qwen2.5 14B-1M · Qwen2.5-VL 72B

Estimates, not guarantees: computed from the official config.json with the same engine as the LLM VRAM Calculator. Real usage depends on your engine version and settings.