Why KL Divergence Is Already Optimal for Knowledge Distillation¶

Counter-Intuitive Finding¶

Confidence-weighted Knowledge Distillation (CW-KD) seems theoretically motivated: give higher weight to tokens where the teacher is more confident. But it is provably redundant.

KL divergence already implements the unique optimal weighting via its built-in \(p_T\) factor — any additional confidence reweighting is mathematically superfluous.

Three Core Theorems¶

Theorem 1: Fisher-Entropy Connection¶

Teacher confidence maps to information-geometric curvature on the probability simplex. This motivates CW-KD from first principles... but the next theorem shows it's unnecessary.

Theorem 2: Score-Function Optimality (KEY RESULT)¶

KL divergence's built-in \(p_T\) factor is the UNIQUE weighting achieving the Cramer-Rao lower bound.

Proof uses: - Cencov's uniqueness theorem (KL is the unique invariant divergence) - Godambe's optimality (optimal estimating function theory)

Implication: Any additive confidence reweighting is mathematically redundant.

Theorem 3: Three-Factor Identity¶

Exact decomposition of the CW-KD vs uniform gap:

• Theory predicts <2.5% difference
• Observed: <0.01% across all experiments

Thermodynamic Conjugacy (de Bruijn Identity)¶

Temperature \(\tau\) maximizing dark-knowledge transfer necessarily destroys the confidence signal that CW-KD relies on.

As temperature increases: - More dark knowledge is exposed (good for KD) - Confidence signal is destroyed (bad for CW-KD) - These are thermodynamically conjugate — you can't have both

Experimental Validation¶

11 Model Configurations, 6 Weighting Strategies¶

Scale Independence (124M to 1T)¶

Practical Takeaway¶

Just use standard KL divergence. Don't waste compute on confidence weighting — the math shows it's already built in.