ATLAS: 跨尺度的长上下文能力全面测试
ATLAS: All-round Testing of Long-context Abilities across Scales
AI总结：
提出ATLAS基准框架，通过分层分类、长度感知AUC评分和ATLAScore聚合指标，系统评估长上下文语言模型在不同长度和任务上的性能退化与能力分布。
摘要：
长上下文语言模型现在宣称上下文窗口可达数百万token，然而评估通常报告单一长度或狭窄的任务族，掩盖了两种失败模式：性能随长度增长而崩溃，以及强大的检索能力不一定能迁移到下游使用。我们提出ATLAS，一个重新定义长上下文评估为长度依赖能力剖析的基准框架。ATLAS贡献了三个方法论原则：(i) 分层分类法，将基础操作与应用工作负载分离，以便归因失败；(ii) 长度感知AUC评分，在固定的8K-1M网格上积分分数-长度曲线，用完整的退化曲线替代单点指标；(iii) ATLAScore，对分类类别进行调和平均聚合，惩罚不平衡的剖面，并通过非线性最终聚合从子集分数进行端到端不确定性传播。我们在八个能力维度上实例化该框架，包含九个可审计组件和6,438个实例，并评估了26个模型。Gemini-3.1-Pro-Preview在128K处领先，Claude-Opus-4.6在1M处领先。排名在ATLASscore@8K-128K和ATLASscore@8K-1M之间大幅重新洗牌：7个模型移动至少两个排名，两个分类层仅共享61%的跨模型方差，个别排名差距高达12位。这些结果支持按能力和长度报告长上下文质量，而不是单一的标题分数。
英文摘要：
Long-context language models now advertise context windows up to millions of tokens, yet evaluations typically report a single length or a narrow task family, masking two failure modes: performance can collapse as length grows, and strong retrieval need not transfer to downstream use. We present ATLAS, a benchmarking framework that redefines long-context evaluation as length-dependent capability profiling. ATLAS contributes three methodological principles:(i) a layered taxonomy separating foundational operations from application workloads so failures can be attributed, (ii) length-aware AUC scoring that integrates score-length curves over a fixed 8K-1M grid, replacing single-point metrics with full degradation profiles, and (iii) ATLAScore, a harmonic-mean aggregate over taxonomy categories that penalizes imbalanced profiles, with end-to-end uncertainty propagation from subset scores through the nonlinear final aggregate. We instantiate the framework across eight capability dimensions with nine auditable components and 6,438 instances, and evaluate 26 models. Gemini-3.1-Pro-Preview leads at 128K, Claude-Opus-4.6 leads at 1M. Rankings reshuffle substantially between ATLASscore@8K-128K and ATLASscore@8K-1M: 7 models move by at least two ranks, and the two taxonomy layers share only 61% of cross-model variance, with individual rank gaps up to 12 positions. These results support reporting long-context quality by capability and length, not by a single headline score.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
