COMPLETED AI/ML

The Recall Ceiling — Why LLM Recommendation Reranking Fails in Practice

Retrieval recall, not reranking sophistication, sets the ceiling on two-stage recommendation. Proved as a combinatorial bound, then falsified against 17 optimisation strategies, 12 retrieval methods, 10 reranker types and 8 datasets across 3 domains. Accepted at CIKM 2026 (main track, oral).

VIEW_SOURCE() DOCS()
status COMPLETED
type AI/ML
started 2025-11-01
completed 2026-08-23
stack Python PyTorch LightGCN GraphSAGE Qwen2.5 FAISS

// DESCRIPTION

Accepted at CIKM 2026 (main track, oral presentation). DOI 10.1145/3799682.3841132 — resolves once the ACM Digital Library publishes the proceedings.

The problem with how rerankers are evaluated

A number of LLM-based recommendation rerankers are evaluated under an oracle protocol that guarantees the ground-truth item is present in the set being scored — either by injecting it into the candidate list, or by scoring it against sampled negatives. Under that protocol the reranker is measured on a problem it will never face in deployment, where the retriever decides what the reranker is allowed to see.

The bound

Two-stage performance is capped by first-stage recall. No amount of reranking can recover an item the retriever did not return, so the achievable metric is bounded combinatorially by the candidate set — a ceiling, not a tendency.

Falsification rather than demonstration

The claim is set up so it could fail, and then attacked: 17 optimisation strategies, 12 retrieval methods, 10 reranker types, 8 datasets, 3 domains. The paper ships a falsification ledger recording what was tried against the bound and what happened, alongside an interactive RAEP diagnostic on the project page.

Reproducibility

The public artifact carries a complete float → script → JSON map: every number in the paper resolves to the script that produced it and the file it was read from.

Two mechanical audits on the finished draft each caught a defect the other missed — an orphaned table cross-reference lost during the page squeeze, and a bibliography entry with a fabricated author list. That second catch is what bibguard exists for.

// HIGHLIGHTS

  • CIKM 2026 main track — accepted as an oral
  • The oracle evaluation protocol measures rerankers on a problem they never face
  • Recall ceiling proved as a combinatorial bound on two-stage performance
  • Survived falsification across 17 strategies × 12 retrievers × 10 rerankers × 8 datasets
  • Public artifact maps every reported float to the script and JSON that produced it
< BACK_TO_PROJECTS()