DeltaCache

Prefix-Trie KV Cache Optimization for LLM Serving

DeltaCache is a KV cache management system for LLM serving that achieves 10-16x faster warm start latency compared to vLLM's default cache management. The core innovation is a prefix trie data structure that indexes cached KV states by their token prefix, enabling O(prefix_length) lookup and maximal cache reuse across requests that share common prefixes (system prompts, few-shot examples, etc.).

The system implements delta compression: instead of storing full KV caches for each unique prefix, it stores only the delta between a new prefix and its closest ancestor in the trie. This dramatically reduces memory consumption, enabling 3-5x more concurrent cache entries in the same GPU memory budget. Combined with INT8 and INT4 quantization of cached KV states, the memory savings compound to 8-15x.

Validation is thorough: 84 end-to-end tests across TinyLlama (1.1B), Mistral-7B, and Qwen2 (7B) models covering diverse prompt patterns (chat, RAG, code completion, agentic tool use). The latency improvements are consistent across all tested scenarios, with the largest gains in high-prefix-overlap workloads like multi-turn chat and RAG pipelines.

DeltaCache is designed as a drop-in replacement for vLLM's cache manager, requiring minimal code changes for integration. The prefix trie operations are lock-free for read paths, ensuring no throughput degradation under concurrent access.

• 10-16x faster warm latency vs vLLM default KV cache management
• Validated across 84 tests on TinyLlama, Mistral-7B, and Qwen2
• Delta compression + INT8/INT4 quantization for 8-15x memory savings
• Lock-free prefix trie for concurrent read access without throughput loss
• Drop-in replacement for vLLM cache manager with minimal integration effort