Serving Llama 3.1 70B with Text Generation Inference on RTX Spark
Production Text Generation Inference deployment of Llama 3.1 70B on RTX Spark. Configs, systemd, TLS, observability.
Install
Install Text Generation Inference via pip/docker/apt as appropriate. Verify GPU visibility with nvidia-smi.
- # Text Generation Inference install
- # see Text Generation Inference docs
Config
Optimized Text Generation Inference config for Llama 3.1 70B at Q4_K_M: batch, KV cache dtype, and parallelism knobs.
Systemd unit
[Unit] Description=Text Generation Inference for Llama 3.1 70B — [Service] ExecStart=/usr/bin/tgi serve llama-3-1-70b --port 8000 — Restart=always.
nginx + TLS
Terminate TLS at nginx with a Let's Encrypt cert, rate-limit at 20 req/s per IP, and proxy to the local Text Generation Inference port.
Observability
Scrape Text Generation Inference Prometheus metrics into Grafana; watch tokens_per_second, batch_size, and kv_cache_usage.
Frequently asked questions
Is Text Generation Inference the fastest for Llama 3.1 70B?
It's the easiest, not always the fastest.
Can I run multiple models with one instance?
One process per model is standard.