Conference
CIKM
Featured
The Recall Ceiling: Why LLM Recommendation Reranking Fails in Practice
Zhaohui Wang
Proceedings of the 35th ACM International Conference on Information and Knowledge Management (CIKM), Rome, Italy, 2026
// ABSTRACT
Some LLM-based recommendation rerankers are evaluated under an oracle protocol that guarantees the ground-truth item is present in the scored set, either by injecting it into the candidate list or by scoring it against sampled negatives. We show that this protocol overestimates realistic performance by 92-95% in NDCG@10 across three primary Amazon datasets, so oracle results do not transfer to deployment. The root cause is a recall ceiling: realistic retrieval covers only 2-19% of relevant items at K=100 across eight datasets in three domains (Amazon products, MovieLens movies, MIND news), placing a deterministic upper bound on any closed-candidate reranker's top-k NDCG (expected NDCG@k cannot exceed recall at the reranker's own candidate-window size under leave-one-out evaluation).
Under realistic retrieval, every optimisation strategy we tested fails to produce a statistically significant improvement over the collaborative-filtering baseline on our primary Amazon datasets: prompt engineering, model scaling across a 168x parameter range, sequential models, three supervised neural rerankers (LambdaMART, RankNet, MLP) under a disjoint train/eval user split, a LoRA-fine-tuned LLaMA-3.2-3B reranker, and a hybrid CF+BM25+Dense retrieval with reciprocal-rank fusion. Adding text-aware retrieval raises recall on Beauty but does not lift end-to-end NDCG. Giving the LLM the upstream CF ranks and scores directly - the fairest setting we can construct - narrows but does not close the gap to CF, and does so by making the LLM adhere more closely to the CF order rather than by adding information.
We propose the Recall-Aware Evaluation Protocol (RAEP) as a diagnostic: classify the regime by retrieval recall, then evaluate reranking only where the ceiling permits differentiation. In the retrieval regimes we can measure (Recall@100 of 2-19%), the path to effective LLM recommendation runs through retrieval improvement rather than reranker sophistication; we do not claim this ordering holds at the higher recall, richer feature sets, and online feedback available to production systems.
Under realistic retrieval, every optimisation strategy we tested fails to produce a statistically significant improvement over the collaborative-filtering baseline on our primary Amazon datasets: prompt engineering, model scaling across a 168x parameter range, sequential models, three supervised neural rerankers (LambdaMART, RankNet, MLP) under a disjoint train/eval user split, a LoRA-fine-tuned LLaMA-3.2-3B reranker, and a hybrid CF+BM25+Dense retrieval with reciprocal-rank fusion. Adding text-aware retrieval raises recall on Beauty but does not lift end-to-end NDCG. Giving the LLM the upstream CF ranks and scores directly - the fairest setting we can construct - narrows but does not close the gap to CF, and does so by making the LLM adhere more closely to the CF order rather than by adding information.
We propose the Recall-Aware Evaluation Protocol (RAEP) as a diagnostic: classify the regime by retrieval recall, then evaluate reranking only where the ceiling permits differentiation. In the retrieval regimes we can measure (Recall@100 of 2-19%), the path to effective LLM recommendation runs through retrieval improvement rather than reranker sophistication; we do not claim this ordering holds at the higher recall, richer feature sets, and online feedback available to production systems.
// BIBTEX
@inproceedings{wang2026recall,
title = {The Recall Ceiling: Why LLM Recommendation Reranking Fails in Practice},
author = {Zhaohui Wang},
booktitle = {Proceedings of the 35th ACM International Conference on Information and Knowledge Management (CIKM), Rome, Italy},
year = {2026},
month = {11},
doi = {10.1145/3799682.3841132},
}