> Incentive-Compatible Prediction Aggregation: Brier vs VCG vs Externality
Comprehensive experiment findings: the original externality mechanism has 7/8 exploitable strategies. Brier Score and VCG fix this. VCG dominates for safety-critical applications. Mechanism design is 4x more data-efficient than neural networks.
// TABLE_OF_CONTENTS
Incentive-Compatible Prediction Aggregation¶
The Incentive Crisis¶
The original externality mechanism (\(u = -\text{Ext}^2\)) is deeply flawed: 7 out of 8 non-truthful strategies can exploit it.
| Strategy | Utility Gain | Impact |
|---|---|---|
| Truthful | baseline | FN=0.0001 |
| Underconfident 10% | +82% | FN=0.0001 |
| Underconfident 50% | +94% | FN=0.0003 |
| Always High | +96% | FN=0.0000 |
| Always Low | +97% | FN=0.0038 |
| Random | +96% | FN=0.0011 |
Root cause: \(u = -\text{Ext}^2\) rewards minimizing influence, not maximizing information.
The Fix: Proper Scoring Rules¶
| Utility Function | Exploits | IC? |
|---|---|---|
| Ext_Squared | 5/6 | No |
| Ext_Linear | 3/6 | No |
| Brier Score | 0/6 | Yes |
| VCG | 0/6 | Yes |
VCG Dominates in Safety¶
Real NSL-KDD dataset results:
| Mechanism | FN Rate | FP Rate |
|---|---|---|
| VCG | 0.305 | 0.031 |
| Ext_Linear | 0.363 | 0.028 |
| Brier Score | 0.387 | 0.028 |
| Majority Vote | 0.380 | — |
VCG wins 7/8 FP-sensitive scenarios.
Data Efficiency: 4x Better than Neural Nets¶
| Samples | VCG (FN) | DeepSets (FN) |
|---|---|---|
| 50 | 0.000 | 0.348 |
| 100 | 0.000 | 0.270 |
| 200 | 0.000 | 0.012 |
| 500 | 0.001 | 0.004 |
VCG achieves perfect FN at 50 samples; DeepSets needs 200.
Communication Robustness¶
| Failure Mode | FN Rate | Degradation |
|---|---|---|
| None | 0.060 | — |
| 50% packet loss | 0.064 | +6.7% |
| 25% corruption | 0.030 | -50% |
Mechanism design is robust to severe communication failures — critical for edge deployment.
Practical Recommendations¶
- Safety-critical → VCG (best FN, IC)
- IC required, less conservative → Brier Score
- Low-data regime → mechanism design over neural nets (4x efficiency)
- Non-strategic agents → externality can work (FN=0.363)
- Edge deployment → mechanism design (robust to failures, scales to 200 agents at <2ms)