> Speculative Recommendation Serving: From LLM Decoding to Caching
RecCache transfers the draft-and-verify paradigm from speculative LLM decoding to recommendation caching: user clusters as draft models, score-ratio acceptance criterion, achieving 1.0-3.8x speedup training-free with O(T*epsilon) regret.
// TABLE_OF_CONTENTS
Speculative Recommendation Serving¶
The Structural Analogy¶
| LLM Speculative Decoding | Speculative Rec Serving |
|---|---|
| Draft model q(x) | Cluster cache C[c] |
| Target model p(x) | Full recommender R(u) |
| min(1, p/q) acceptance | Score-ratio acceptance |
| Multi-token speculation | Multi-cluster speculation |
Results¶
| Dataset | NDCG@10 | Accept Rate | Speedup |
|---|---|---|---|
| ML-1M | .091 | 91% | 2.2x |
| A-Electronics | .010 | 59% | 3.1x |
| MIND | .014 | 32% | 3.8x |
MIND: NDCG exactly matches fresh recommendations at 3.8x speedup.
Multi-Cluster Gain¶
49-68% of accepted recommendations come from non-nearest clusters. Single-cluster caching would substantially reduce coverage.
Training-Free Regret¶
\(O(T \cdot \epsilon)\) regret without training vs \(O(\sqrt{T})\) for RL-based caching (CARL). RecCache matches CARL's NDCG at zero training cost.
Methodology Fix¶
Critical correction: early experiments didn't exclude training history items from recommendations, inflating NDCG. The fix reduces absolute numbers but ensures genuine generalization — a common pitfall in recommendation evaluation.