> ProbML Submission Journey: From Draft to 31-Page Archival Paper

LLM Routing 高级 2026-04-01 06:41 2026-04-01
#ProbML #submission #paper writing #reference verification #experiment pipeline

The full journey of submitting the HMM-TS routing paper to ProbML 2026: from initial KDD draft to 31-page archival paper with 38 verified references, 15K real API queries, and 6 benchmarks.

// TABLE_OF_CONTENTS

ProbML 2026 Submission Journey

Timeline

Date Milestone
Mar 1 Project initialization, HMM routing concept
Mar 12 Initial KDD 2026 draft with robustness analysis
Mar 15 Complete ProbML draft (9 sections + 10 appendices)
Mar 17 Terminology audit: renamed all commercial model names to open-source identifiers
Mar 19 Sequential advantage experiments (3 new experiment tracks)
Mar 20 Submitted to ProbML 2026 Proceedings Track

Key Evolution Points

From Proxy to Real API Quality

The biggest upgrade was replacing proxy quality scores with real API evaluations: - 15,000 queries across 3 datasets (OASST, StackOverflow, MBPP) - Multi-provider pipeline: Groq paid (generation) + Llama-3.3-70b (judge) - Oracle quality: OASST 99%, StackOverflow 94%, MBPP 92% - Bug fixes discovered: context dimension mismatch, Cohen's d calculation, seed independence

Sequential Advantage Experiments

Three new experiments specifically validating the HMM temporal advantage:

  1. Sequential vs Shuffled: HMM-TS regret 41.1 vs LinUCB 45.8 (p=0.0007, Cohen's d=1.04)
  2. Session Length Sweep: Advantage grows from 1.15x (L=3) to 1.30x (L=50)
  3. Multi-Model (K>2): Advantage grows from 1.12x (K=2) to 1.20x (K=5)

Reference Verification

All 38 bibliography entries manually verified. Key fixes: - Le Gland DOI corrected - FrugalGPT year updated to 2024 - All model references changed from commercial to open-source names

Paper Polish

  • Title shortened to "Bayesian Belief Compression for Sequential Model Selection"
  • Abstract reframed: emphasis on theoretical contribution over empirical claims
  • Table formatting unified: all tables now consistent tabular* width
  • Sequential advantage promoted from appendix to main text (S7.5)
  • 31 pages, 14 figures, 5 tables — clean compile with zero warnings

Experiment Infrastructure

The cached oracle system is the backbone:

cache/
├── oracle_real_oasst.json       # 5000 queries, real API quality
├── oracle_real_stackoverflow.json
├── oracle_real_mbpp.json
├── oasst/                       # Dataset-specific caches
├── stackoverflow/
└── mbpp/

This enables reproducible experiments without re-querying expensive APIs. Total cache: ~15K real API quality scores at ~$5 total API cost.

Lessons Learned

  1. Real API > Proxy: Reviewers can tell when quality scores are synthetic. The $5 investment in real API oracles was worth it.
  2. Reference verification is non-negotiable: 3 of 38 references had errors. A single fabricated reference can sink a paper.
  3. Sequential advantage requires careful experimental design: Simply running on sequential data isn't enough — need controlled shuffled comparison to isolate temporal effects.
< BACK_TO_NOTES()