EBM-Market

The Problem: Machine Learning Assumes Anomaly Equals Risk — But Does It?

Virtually every machine learning system for risk prediction — fraud detection, market crash forecasting, traffic incident prediction — is built on the same implicit assumption: unusual events are risky events. If a transaction looks different from normal transactions, flag it as fraud. If a market day looks different from normal market days, treat it as elevated risk. If a traffic pattern deviates from baseline, predict an incident. This anomaly-equals-risk assumption is so pervasive that it is rarely stated explicitly, let alone tested empirically.

EBM-Market challenges this assumption head-on. The motivating observation is this: in financial markets, the highest-density trading days — days that look most 'normal' — frequently precede major crashes. Herding behavior causes markets to appear extremely stable and 'normal' in the weeks before a collapse. Anomaly-based detectors would flag these pre-crash days as low risk precisely because they look like everything else. Yet empirically, these are the most dangerous days in the dataset. If this is true — if density and risk are positively correlated in some domains — then the entire anomaly-detection paradigm for risk prediction is backwards for those domains.

问题：机器学习假设异常等于风险——但真的如此吗？
几乎所有风险预测ML系统都建立在同一隐含假设上：不寻常事件就是风险事件。这一假设如此普遍，以至于鲜少被明确陈述，更少被实证检验。EBM-Market正面挑战这一假设。核心观察：在金融市场中，看起来最“正常”的高密度交易日往往是重大崩盘前的征兆——羊群效应使市场在崩溃前数周显得极度稳定。异常检测器会将这些崩盘前的日子标记为低风险，恰恰因为它们看起来和其他日子一样。如果密度与风险在某些领域正相关，那么整个异常检测范式对那些领域来说就是颠倒的。

Situation: Three Domains, Three Different Risk Topologies

EBM-Market tests the density-risk hypothesis across three empirically distinct domains:

• Financial markets: daily OHLCV data from FRED and Yahoo Finance, covering equities, bonds, and cryptocurrency markets across multiple market regimes
• Traffic flow: inD dataset (intersection drone footage), covering vehicle trajectory density and incident occurrence at monitored intersections
• Multi-agent reinforcement learning: SMAC v2 (StarCraft Multi-Agent Challenge), measuring coordination breakdown risk as a function of agent state density

These three domains were chosen because they represent qualitatively different risk topologies: financial markets are hypothesized to show density=risk (herding), traffic is hypothesized to show a mixed signal, and MARL recommendations are hypothesized to invert the relationship.

场景：三个领域，三种不同的风险拓扑
EBM-Market在三个实证截然不同的领域检验密度-风险假设：金融市场（FRED/Yahoo Finance每日OHLCV数据，涵盖股票、债券和加密货币）、交通流（inD数据集，交叉路口无人机轨迹）、多智能体强化学习（SMAC v2，协调崩溃风险作为智能体状态密度的函数）。这三个领域代表质量上不同的风险拓扑，分别假设呈现密度=风险、混合信号和关系反转。

Task: Formally Test Whether Density Predicts Risk Better Than Anomaly

The research question is formalized as: does a density-based risk score (derived from a fitted probability density model) outperform anomaly-based risk scores (isolation forest, one-class SVM, autoencoder reconstruction error) on the task of predicting adverse outcomes? The comparison must be statistically rigorous, accounting for temporal autocorrelation, regime changes, and the look-ahead bias that plagues financial ML evaluation.

任务：正式检验密度是否比异常更能预测风险
研究问题形式化为：从拟合概率密度模型导出的密度风险分数，是否优于基于异常的风险分数（隔离森林、单类SVM、自编码器重建误差）在预测不利结果方面？比较必须统计上严格，考虑时序自相关、机制变化和金融ML评估中的前瞻偏差。

Action: GMM Density + 10K Permutation Tests + Walk-Forward Evaluation

The density model uses a Gaussian Mixture Model (GMM) fitted to the feature representation of each domain. The number of mixture components is selected via BIC on held-out data. The fitted GMM density score is then used directly as a risk signal: high density = high risk (for domains where the hypothesis holds) or low density = high risk (for domains where it inverts).

Statistical validation uses 10,000 permutation tests on the density-risk correlation, breaking temporal structure and resampling to establish null distributions. Walk-forward cross-validation (expanding window, no data leakage) is used for financial evaluation to approximate real deployment conditions. The Density Risk Ratio (DRR) is introduced as a summary statistic: DRR > 1 indicates density-risk alignment (high density leads to high risk), DRR < 1 indicates inversion, DRR = 1 is null (no relationship).

Density Risk Ratio (DRR) Framework

  Domain            |  DRR   |  Interpretation
  ------------------+--------+----------------------------------
  Finance (equities)|  1.96x |  High density -> 96% more risk
  Traffic (inD)     |  1.14x |  Mild density-risk alignment
  MARL (SMAC v2)    |  0.59x |  INVERTED: density = safety
  ------------------+--------+----------------------------------

  AUROC comparison:
  GMM Density:  0.931 - 0.954  (finance + traffic)
  Best Anomaly: 0.802          (isolation forest)
  Walk-forward: ~0.50 (near-random) -- honest degradation reported

行动：GMM密度 + 10000置换检验 + 滚动前向评估
密度模型使用高斯混合模型（GMM），BIC选择混合成分数，拟合密度分数直接用作风险信号。统计验证采用10000次置换检验，建立零分布。金融评估使用扩展窗口前向交叉验证（无数据泄漏）。引入密度风险比（DRR）作为汇总统计量：DRR>1表示密度-风险对齐，DRR<1表示反转。

Results: AUROC 0.931-0.954 vs. 0.802; Honest Walk-Forward Degradation Reported

• AUROC 0.931-0.954 for GMM density-based risk in financial and traffic domains, vs. the best anomaly baseline at 0.802 (isolation forest)
• Density Risk Ratio 1.96x in finance: days in the highest density quartile had 96% higher realized risk than the lowest density quartile
• Density Risk Ratio 1.14x in traffic: mild but statistically significant density-risk alignment (p < 0.001 by permutation test)
• INVERTED in MARL recommendations: DRR = 0.59x — high-density agent states are significantly safer, meaning anomaly-based detectors would incorrectly flag safe states as risky and miss genuinely dangerous rare states
• Walk-forward evaluation degrades to near-random (AUROC ~0.50) — an honest limitation reported transparently: the static density signal does not survive realistic deployment simulation, suggesting caution for live trading
• 10,000 permutation tests confirm all cross-domain results at p < 0.001

Accepted: From Risk Scoring to Risk Allocation (SRAS @ ICML 2026)

The allocation half of this work was accepted at the SRAS workshop (Scalable and Resilient Agentic Systems) at ICML 2026, where all three reviews are public. It reframes monitoring from per-state scoring to combinatorial allocation: given a fragility score, choosing what to monitor is a subset-selection problem over interdependent states, encoded as a QUBO with a λ-controlled risk–diversity tradeoff. The resulting Pareto frontier contains greedy top-K, MMR and k-DPP as fixed operating points, which organises a fragmented literature rather than adding to it. The gain over greedy grows with scale (+24% at n=15 to +66% at n=200), a learned λ policy reaches 99.5% of an oracle grid search at no inference cost, and the same QUBO instances execute unmodified on Rigetti superconducting QPUs (Ankaa-3 and Cepheus-1-108Q via Amazon Braket) — reported as a compatibility property of the formulation, not a quantum-advantage claim. The main-track paper, which adds a formal definition of the risk event and confidence intervals on a small positive class, is under revision for PAKDD 2027.

结果：AUROC 0.931-0.954 vs. 0.802；诚实的前向滚动退化
关键指标：金融和交通领域GMM密度风险的AUROC为0.931-0.954（最优异常基线为0.802）；金融领域DRR=1.96x，交通领域DRR=1.14x，MARL推荐领域DRR=0.59x（关系反转）；滚动前向评估诚实地揭示AUROC退化至~0.50（近随机），对实盘交易需谨慎；10000次置换检验所有结果p<0.001。该工作的「分配」部分已被 ICML 2026 SRAS workshop（Scalable and Resilient Agentic Systems）录用，三份评审全部公开：把监控从逐状态打分重构为组合分配，编码为 QUBO，其 Pareto 前沿把 greedy top-K、MMR 与 k-DPP 都容纳为固定工作点；相对 greedy 的增益随规模上升（n=15 时 +24%，n=200 时 +66%），同一批 QUBO 实例可不加修改地在 Rigetti 超导 QPU 上执行——这被明确报告为形式化的兼容性，而非量子优势主张。补充了风险事件形式化定义与小正类置信区间的主会论文正在为 PAKDD 2027 修改。

• Accepted at the SRAS workshop, ICML 2026 (Scalable and Resilient Agentic Systems) — challenges the anomaly=risk assumption in ML risk prediction
• AUROC 0.931–0.954 (GMM density) vs. 0.802 (best anomaly baseline) in finance and traffic
• Density Risk Ratio 1.96x in financial markets — highest-density days precede crashes (herding effect)
• Density Risk Ratio 1.14x in traffic (inD dataset) — mild but significant density-risk alignment
• INVERTED in MARL (DRR = 0.59x) — density signals safety in agent coordination; anomaly detectors are backwards
• Honest limitation reported: walk-forward evaluation degrades to AUROC ~0.50 — no overfitting to favorable evaluation
• 10,000 permutation tests, walk-forward cross-validation — rigorous against temporal autocorrelation and look-ahead bias
• Multi-domain validation: FRED/Yahoo Finance equities+crypto, inD traffic, SMAC v2 MARL across 3 risk topologies
• Risk allocation, not risk scoring — monitoring recast as combinatorial subset selection and encoded as a QUBO whose Pareto frontier contains greedy top-K, MMR and k-DPP as fixed operating points
• +24% to +66% diversity over greedy as the candidate pool grows from n=15 to n=200; a learned λ policy reaches 99.5% of an oracle grid search at no inference cost
• The same QUBO instances run unmodified on Rigetti superconducting QPUs (Ankaa-3, Cepheus-1-108Q via Amazon Braket) — a compatibility property of the formulation, not a quantum-advantage claim