误差图与误差图谱：大型语言模型失败景观的绘制
ErrorMap and ErrorAtlas: Charting the Failure Landscape of Large Language Models
AI总结：
ErrorMap和ErrorAtlas通过分析LLM失败原因，揭示模型弱点，为模型改进和评估提供更深入的洞察。
摘要：
大型语言模型（LLM）基准测试告诉我们模型何时失败，但不能解释为何失败。在推理数据集上得到错误答案可能源于格式问题、计算错误或数据集噪声，而非推理能力不足。在没有分离这些原因的情况下，基准测试仍然不完整，无法可靠地指导模型改进。我们介绍了ErrorMap，这是首个绘制LLM失败来源的方法。它提取模型的唯一“失败签名”，明确基准测试测量什么，并扩展错误识别以减少盲区。这有助于开发者调试模型，使基准目标与结果一致，并支持有根据的模型选择。ErrorMap在任何模型或数据集上都采用相同的逻辑。应用我们的方法到35个数据集和83个模型，我们生成ErrorAtlas，一个错误类型的分类学，揭示了反复出现的失败模式。ErrorAtlas突出了目前在LLM研究中较少探索的错误类型，如输出中遗漏必需细节和问题误解。通过将关注点从模型成功的地方转向为何失败，ErrorMap和ErrorAtlas使高级评估成为可能——一种能够暴露隐藏弱点并引导进步的评估。与通常通过任务级指标衡量的成功不同，我们的方法引入了更深层次的评估层，可以应用于所有模型和任务，提供更丰富的模型行为和限制的见解。我们公开了分类学和代码，并计划定期更新ErrorAtlas，随着新基准和模型的出现。
英文摘要：
Large Language Models (LLM) benchmarks tell us when models fail, but not why they fail. A wrong answer on a reasoning dataset may stem from formatting issues, calculation errors, or dataset noise rather than weak reasoning. Without disentangling such causes, benchmarks remain incomplete and cannot reliably guide model improvement. We introduce ErrorMap, the first method to chart the sources of LLM failure. It extracts a model's unique "failure signature", clarifies what benchmarks measure, and broadens error identification to reduce blind spots. This helps developers debug models, aligns benchmark goals with outcomes, and supports informed model selection. ErrorMap works on any model or dataset with the same logic. Applying our method to 35 datasets and 83 models we generate ErrorAtlas, a taxonomy of model errors, revealing recurring failure patterns. ErrorAtlas highlights error types that are currently underexplored in LLM research, such as omissions of required details in the output and question misinterpretation. By shifting focus from where models succeed to why they fail, ErrorMap and ErrorAtlas enable advanced evaluation - one that exposes hidden weaknesses and directs progress. Unlike success, typically measured by task-level metrics, our approach introduces a deeper evaluation layer that can be applied globally across models and tasks, offering richer insights into model behavior and limitations. We make the taxonomy and code publicly available with plans to periodically update ErrorAtlas as new benchmarks and models emerge.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
