How-toUpdated 2026-06-28 · 12 min read · by RTXsparks Lab
How to Run Llama-3.1-70B on RTX Spark (Fast & Reliable)
Step-by-step guide to running Llama-3.1-70B locally on RTX Spark with vLLM and NVFP4 quantization. Includes benchmarks and troubleshooting.
Install vLLM 0.9 with NVFP4
Run `pipx install 'vllm[nvfp4]==0.9.*'` then `vllm serve meta-llama/Llama-3.1-70B-Instruct --quantization nvfp4 --tensor-parallel-size 1 --max-model-len 32768`.
Benchmark and tune
Use `vllm bench serve` with 8 concurrent clients. Enable `--enable-chunked-prefill` for latency, disable for max throughput.
Frequently asked questions
Does 64GB Spark work for 70B?
Only with GGUF Q3_K_M at ~14 tok/s. We recommend 128GB for 70B production use.