How-toUpdated 2026-07-01 · 9 min read · by RTXsparks Lab
Serving Llama 3.2 3B with vLLM on RTX Spark
Production vLLM deployment of Llama 3.2 3B on RTX Spark. Configs, systemd, TLS, observability.
Install
Install vLLM via pip/docker/apt as appropriate. Verify GPU visibility with nvidia-smi.
- # vLLM install
- pip install vllm
Config
Optimized vLLM config for Llama 3.2 3B at Q6_K: batch, KV cache dtype, and parallelism knobs.
Systemd unit
[Unit] Description=vLLM for Llama 3.2 3B — [Service] ExecStart=/usr/bin/vllm serve llama-3-2-3b --port 8000 — Restart=always.
nginx + TLS
Terminate TLS at nginx with a Let's Encrypt cert, rate-limit at 20 req/s per IP, and proxy to the local vLLM port.
Observability
Scrape vLLM Prometheus metrics into Grafana; watch tokens_per_second, batch_size, and kv_cache_usage.
Frequently asked questions
Is vLLM the fastest for Llama 3.2 3B?
Yes for throughput; TensorRT-LLM edges it on p50 latency.
Can I run multiple models with one instance?
Yes with --served-model-name and multi-LoRA.