Models / Llama
Can an RTX 3060 12GB run Llama 3.2 3B?
Small dense Llama. Comfortable on 8 GB at Q8.
- Stored parameters
- 3B
- Active parameters
- Dense
- Q4 total (estimate)
- 3.4 GB
- License
- Llama 3.2 Community License
Model card. Meta lists Llama 3.2 3B as a 3B model.
Compatibility
| Hardware | Q4 | Q8 | FP16 |
|---|---|---|---|
| M3 Pro 18GB Apple · 18 GB unified | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run | |
| M3 Ultra 512GB Apple · 512 GB unified | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| 2× NVIDIA GB10 NVIDIA · 256 GB unified | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| M3 Ultra 256GB Apple · 256 GB unified | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| M5 Ultra 256GB Apple · 256 GB unified | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| M2 Ultra 192GB Apple · 192 GB unified | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| NVIDIA GB10 NVIDIA · 128 GB unified | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| Ryzen AI Max+ 395 128GB AMD · 128 GB unified | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| M1 Ultra 128GB Apple · 128 GB unified | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| M5 Max 128GB Apple · 128 GB unified | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| 4× RTX 3090 NVIDIA · 96 GB VRAM | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| RTX Pro 6000 Blackwell NVIDIA · 96 GB VRAM | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| M5 Ultra 96GB Apple · 96 GB unified | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| Ryzen AI Max+ 395 64GB AMD · 64 GB unified | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| 2× RTX 3090 NVIDIA · 48 GB VRAM | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| M4 Pro 48GB Apple · 48 GB unified | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| M4 Max 36GB Apple · 36 GB unified | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| M5 Max 36GB Apple · 36 GB unified | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| RTX 5090 NVIDIA · 32 GB VRAM | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| Radeon AI Pro R9700 AMD · 32 GB VRAM | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| Intel Arc Pro B70 Intel · 32 GB VRAM | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| M4 32GB Apple · 32 GB unified | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| M6 32GB Apple · 32 GB unified | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| RTX 3090 NVIDIA · 24 GB VRAM | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| RTX 4090 NVIDIA · 24 GB VRAM | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| TITAN RTX NVIDIA · 24 GB VRAM | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| Radeon RX 7900 XTX AMD · 24 GB VRAM | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| M4 Pro 24GB Apple · 24 GB unified | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| RTX 4060 Ti 16GB NVIDIA · 16 GB VRAM | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| RTX 5060 Ti 16GB NVIDIA · 16 GB VRAM | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| RTX 5070 Ti NVIDIA · 16 GB VRAM | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| M4 16GB Apple · 16 GB unified | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| RTX 3060 12GB NVIDIA · 12 GB VRAM | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| RTX 3080 Ti NVIDIA · 12 GB VRAM | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| RTX 4070 Ti NVIDIA · 12 GB VRAM | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Fits estimate · KV at 8k No measured run |
| RTX 2060 Super NVIDIA · 8 GB VRAM | 3.4 GB Fits estimate · KV at 8k No measured run | 4.8 GB Fits estimate · KV at 8k No measured run | 7.6 GB Tight estimate · KV at 8k No measured run |
Measured runs
| Hardware | Quant | tok/s | Runtime | Source |
|---|---|---|---|---|
| RTX 4090 | not stated | 334.67 | llama.cpp | Source · checked · prefill 12746 |
| RTX 5090 | not stated | 271.4 | llama.cpp | Source · checked |
| RTX 3080 Ti | not stated | 263.24 | llama.cpp | Source · checked · prefill 6476 |
| RTX 4090 | not stated | 237.05 | llama.cpp | Source · checked |
| RTX 4070 Ti | not stated | 196.15 | llama.cpp | Source · checked · prefill 6542 |
| Radeon AI Pro R9700 | not stated | 195.91 | llama.cpp | Source · checked · prefill 3570 |
| M3 Ultra 512GB | not stated | 154.6 | llama.cpp | Source · checked · prefill 2915 |
| M1 Ultra 128GB | not stated | 127.6 | llama.cpp | Source · checked · prefill 1912 |
| RTX 3060 12GB | not stated | 122.85 | llama.cpp | Source · checked · prefill 2801 |
| M3 Pro 18GB | Q4_K_M (2.02 GB, Ollama tag llama3.2:3b) | 55.6 | Ollama 0.40.13 (Metal) | Source · checked |
| M4 32GB | not stated | 41.31 | llama.cpp | Source · checked |