IntegriRef

The Problem: Citation Verification Has Stalled at Layer Zero

Academic integrity tools today can tell you whether a cited paper exists. They check DOIs, verify author names, confirm publication years. This is L0 verification — existence checking. But the most damaging form of citation misconduct is not citing a nonexistent paper; it is citing a real paper to support a claim the paper never makes. A researcher might cite a study that found "no significant effect" to support the sentence "prior work confirms the effect". The DOI is valid, the paper is real, the citation is semantically fraudulent.

No existing system at the time of this project's inception verified what a citation actually claims versus what the cited source actually says. The gap between citation existence and citation semantic fidelity is where most academic misconduct lives, and it is entirely unchecked by current tools.

问题：当前学术引用验证工具止步于L0层——仅核查引用文献是否存在（DOI有效性、作者姓名等）。最具破坏性的引用不当行为并非引用不存在的文献，而是引用真实文献却歪曲其主张。目前没有任何系统验证引用声明与原文内容的语义一致性，这是学术不端行为最集中的地带。

Situation & Task: Five-Layer Semantic Verification Stack

IntegriRef is targeting submission to JASIST (Journal of the Association for Information Science and Technology) and ACL (Association for Computational Linguistics). The project's scope is ambitious: build a complete citation integrity system covering the full verification spectrum from raw existence to probabilistic risk scoring, with production-grade throughput and coverage across 62 academic registries spanning 6 domains (CS, Medicine, Physics, Law, Economics, Humanities).

The five layers are not independent checks but a staged pipeline where each layer is activated only when the previous layer passes, allowing cheap checks to filter out clearly valid references before expensive semantic analysis.

任务：IntegriRef目标投稿JASIST和ACL。项目构建了一套完整的五层引用完整性验证系统，覆盖从原始存在性验证到概率风险评分的完整验证链，支持62个学术注册表、6个领域（计算机科学、医学、物理、法律、经济学、人文学科），并具备生产级处理吞吐量。

Innovation: The Five-Layer Verification Architecture

L0 — Existence Verification: Basic bibliographic validation against 62 registry adapters (CrossRef, PubMed, ArXiv, SSRN, PhilPapers, etc.). Async batch processing achieves 336,000 references per second throughput through connection pooling and Redis caching of registry responses. 14.5x speedup over synchronous baseline.

L1 — Citation Intent Classification: A fine-tuned SciBERT model (86.0% F1) classifies whether each citation is used for background, method, result, or contrast. Intent mismatches (citing a methodology paper to support a result claim) are flagged as anomalies. SciBERT's scientific domain pretraining gives it significantly better performance than general-purpose BERT on academic text.

L2 — Natural Language Inference: The most technically demanding layer. A fine-tuned DeBERTa-v3 model determines whether the citing sentence's claim is entailed, neutral, or contradicted by the cited abstract. DeBERTa-v3 achieves 93.5% accuracy on a human-annotated citation claim dataset, surpassing the human expert agreement baseline of 89.1%. This is the first time an automated system has exceeded human performance on this task.

L3 — Graph Anomaly Detection: The CIDRE algorithm analyzes the citation graph for structural anomalies — citation rings, self-citation clusters, and anomalous connectivity patterns that suggest manipulative citation behavior. Graph-level signals complement sentence-level NLI by catching cases where individual citations look clean but the citation network is suspicious.

L4 — Bayesian Risk Scoring: An 18-signal Bayesian risk model aggregates outputs from L0–L3 plus metadata features (retraction history, author self-citation rate, journal reputation, claim specificity) into a single calibrated risk score. The score is interpretable: each of the 18 signals has a human-readable explanation in the output report.

创新：五层验证架构。L0存在验证：62个注册表适配器，异步批处理实现336K引用/秒（14.5x加速）。L1引用意图分类：SciBERT精调（F1=86.0%），识别背景/方法/结论/对比引用。L2自然语言推理：DeBERTa-v3精调，准确率93.5%，超越人类专家一致性基准（89.1%），这是首次自动化系统在此任务上超越人类表现。L3图异常检测：CIDRE算法分析引用网络结构异常。L4贝叶斯风险评分：18信号贝叶斯模型将L0-L3输出及元数据汇总为可解释风险评分。

Approach: Engineering for Scale and Reliability

The system is built for production deployment, not just research prototyping. 480+ unit and integration tests cover the full pipeline. ONNX quantization reduces DeBERTa-v3 inference latency by 3.2x, making the L2 layer practical for batch processing of large reference lists. Redis caching with TTL-based expiry handles the registry query volume without hammering external APIs.

The 62 registry adapters use a plugin architecture: each adapter implements a common interface, and new registries can be added without modifying core pipeline logic. Domain-specific adapters handle quirks like preprint versioning (ArXiv), clinical trial registration (ClinicalTrials.gov), and legal citation formats (Westlaw, LexisNexis-compatible).

OpenAlex, an open scholarly graph with 250M+ works, serves as the primary metadata backbone, enabling citation graph construction for L3 without requiring commercial database access.

工程实现：480+单元和集成测试覆盖完整流水线。ONNX量化将DeBERTa-v3推理延迟降低3.2x。Redis缓存（TTL过期机制）处理注册表查询量。62个注册表适配器采用插件架构，新注册表可在不修改核心逻辑的情况下接入。OpenAlex（250M+文献开放学术图谱）作为主要元数据骨干，支持无需商业数据库访问的L3引用图构建。

Results and Significance

The headline result — 93.5% NLI accuracy surpassing human expert agreement (89.1%) — represents a qualitative threshold. Below human performance, automated citation checking would produce too many false positives to be trusted; above it, the system can serve as a primary reviewer with humans as the escalation path for borderline cases.

The 336K references/second throughput means a 10,000-reference literature review completes in under 30 milliseconds at L0, enabling real-time checking integrated into manuscript submission systems. The 14.5x async speedup over synchronous processing is achieved through concurrent aiohttp sessions with circuit breakers for registry fault tolerance.

IntegriRef fills the gap between citation existence checking (what current tools do) and full semantic integrity verification (what the research community needs), directly addressing a high-priority concern for publishers, funders, and institutional integrity offices worldwide.

结果与意义：L2 NLI准确率93.5%超越人类专家一致性（89.1%），这是一个质性门槛——只有超越人类水平，自动化引用核查才能作为主要审查者被信任。336K引用/秒的吞吐量意味着10,000篇参考文献的文献综述在L0层30毫秒内完成，可实现嵌入稿件提交系统的实时核查。IntegriRef填补了现有工具（引用存在性检查）与研究界真实需求（语义完整性验证）之间的空白。

• First system to surpass human performance on citation semantic verification: DeBERTa-v3 NLI at 93.5% vs human expert agreement 89.1%
• 5-layer verification stack: L0 existence → L1 intent → L2 NLI → L3 graph anomaly → L4 Bayesian risk (18 signals)
• 336,000 references/second async throughput, 14.5x speedup over synchronous baseline
• 62 registry adapters across 6 academic domains (CS, Medicine, Physics, Law, Economics, Humanities)
• SciBERT intent classifier: 86.0% F1 across 4 citation intent categories
• ONNX quantization reduces DeBERTa-v3 inference latency 3.2x for batch processing
• 480+ tests; OpenAlex (250M+ works) backbone for citation graph construction
• Targeting JASIST and ACL for submission