Serving DeepSeek R1 Distill 70B with Triton Inference Server on RTX Spark
Production Triton Inference Server deployment of DeepSeek R1 Distill 70B on RTX Spark. Configs, systemd, TLS, observability.
Install
Install Triton Inference Server via pip/docker/apt as appropriate. Verify GPU visibility with nvidia-smi.
- # Triton Inference Server install
- # see Triton Inference Server docs
Config
Optimized Triton Inference Server config for DeepSeek R1 Distill 70B at Q4_K_M: batch, KV cache dtype, and parallelism knobs.
Systemd unit
[Unit] Description=Triton Inference Server for DeepSeek R1 Distill 70B — [Service] ExecStart=/usr/bin/triton serve deepseek-r1-distill-70b --port 8000 — Restart=always.
nginx + TLS
Terminate TLS at nginx with a Let's Encrypt cert, rate-limit at 20 req/s per IP, and proxy to the local Triton Inference Server port.
Observability
Scrape Triton Inference Server Prometheus metrics into Grafana; watch tokens_per_second, batch_size, and kv_cache_usage.
Frequently asked questions
Is Triton Inference Server the fastest for DeepSeek R1 Distill 70B?
It's the easiest, not always the fastest.
Can I run multiple models with one instance?
One process per model is standard.