Embedding service with DSPy on RTX Spark

Architecture

DSPy coordinates tool calls, memory, and retrieval. vLLM serves the model locally at http://localhost:8000/v1. Data never leaves the Spark.

Setup

Install DSPy, vLLM, and a vector store (Qdrant recommended). Wire DSPy's LLM adapter to the local endpoint.

• pip install dspy
• vllm serve qwen3-32b --quantization q4_k_m
• docker run -p 6333:6333 qdrant/qdrant

Reference implementation

~150 lines of Python: ingestion, retrieval, tool schemas, and the DSPy graph/crew.

Production checklist

Add tracing (Langfuse or Phoenix), rate limiting, and a fallback route to a smaller model when p95 breaches SLO.

Cost

~$0.02 per 1M tokens amortized. Compare to cloud GPT-class equivalents at $3–15 per 1M.