Serving Qwen2.5-VL 72B with TensorRT-LLM on RTX Spark
Production TensorRT-LLM deployment of Qwen2.5-VL 72B on RTX Spark. Configs, systemd, TLS, observability.
Install
Install TensorRT-LLM via pip/docker/apt as appropriate. Verify GPU visibility with nvidia-smi.
- # TensorRT-LLM install
- # see TensorRT-LLM docs
Config
Optimized TensorRT-LLM config for Qwen2.5-VL 72B at Q4_K_M: batch, KV cache dtype, and parallelism knobs.
Systemd unit
[Unit] Description=TensorRT-LLM for Qwen2.5-VL 72B — [Service] ExecStart=/usr/bin/tensorrt-llm serve qwen2-5-vl-72b --port 8000 — Restart=always.
nginx + TLS
Terminate TLS at nginx with a Let's Encrypt cert, rate-limit at 20 req/s per IP, and proxy to the local TensorRT-LLM port.
Observability
Scrape TensorRT-LLM Prometheus metrics into Grafana; watch tokens_per_second, batch_size, and kv_cache_usage.
Frequently asked questions
Is TensorRT-LLM the fastest for Qwen2.5-VL 72B?
Yes for interactive latency; vLLM ties on throughput.
Can I run multiple models with one instance?
One process per model is standard.