> Fisher Information vs Entropy Weighting: A Code-Paper Mismatch Story

Knowledge Distillation 高级 2026-03-31 08:43 2026-03-31

#Fisher information #entropy #knowledge distillation #code review #theory-practice gap

Discovery and resolution of a critical code-paper mismatch: paper claims layer-level Fisher weighting, code implements token-level entropy weighting. Both have theoretical basis, connected through the exponential family identity F = -nabla^2 H.

// TABLE_OF_CONTENTS

Fisher Information vs Entropy Weighting: Code-Paper Mismatch

Fisher Information vs Entropy Weighting: Code-Paper Mismatch¶

The Discovery¶

A critical mismatch between paper and code:

Aspect	Paper Claims	Code Implements
Weighting target	Layer	Token
Computation basis	Parameter gradients	Output entropy
Theoretical source	Fisher information matrix	Shannon entropy
Physical meaning	Parameter importance	Prediction confidence

What the code actually does:¶

t_p = F.softmax(t_logits / self.temp, dim=-1)
ent = -(t_p * torch.log(t_p + 1e-8)).sum(dim=-1)
max_ent = np.log(t_logits.size(-1))
fw = (1.0 - ent / max_ent).clamp(min=0.1, max=1.0)  # confidence weight

This is token-level entropy weighting, not Fisher information weighting!

The Deep Connection¶

Despite the mismatch, Fisher information and entropy ARE mathematically connected.

Exponential Family Identity¶

For exponential family distributions $p(x|\theta)$:

\[F(\theta) = \nabla^2 A(\theta) = -\nabla^2 H(p_\theta)\]

Fisher information IS the negative Hessian of entropy.

For Softmax Outputs¶

Fisher information of softmax w.r.t. logits:

\[F_{z_i z_j} = \frac{1}{\tau^2}(p_i \delta_{ij} - p_i p_j)\]

The Fisher trace:

\[\text{tr}(F) = \frac{1}{\tau^2}\sum_i p_i(1-p_i) = \frac{1}{\tau^2}(1 - \sum_i p_i^2)\]

where $\sum p_i^2$ is related to Renyi entropy. So: low Shannon entropy ↔ high confidence ↔ low output Fisher information ↔ more stable output space.

The Confidence Weight as Information Gain¶

The implemented weight: $$w_t = 1 - \frac{H(p_t)}{H_{max}} = \frac{\log V - H(p_t)}{\log V}$$

This is normalized negative entropy = information gain relative to maximum entropy. Properties: - $w_t \in [0, 1]$ - Teacher fully certain → $w_t = 1$ - Teacher fully uncertain → $w_t = 0$

Resolution¶

Both approaches have valid theoretical foundations: 1. Token-level (entropy): Information-theoretic — low-entropy tokens = reliable knowledge 2. Layer-level (Fisher): Parameter importance — high Fisher = important layers

The paper was updated to correctly describe the token-level approach with information-geometric justification, maintaining theoretical honesty while preserving the valid connection to Fisher information theory.

Lesson Learned¶

Always verify code matches paper claims before submission. The theoretical analysis of the mismatch itself became a valuable contribution — revealing the Fisher-entropy connection for knowledge distillation.

< BACK_TO_NOTES()