大型语言模型真的能理解当先例被推翻时的情况吗？
Do LLMs Truly Understand When a Precedent Is Overruled?
AI总结：
本文通过评估LLMs在识别美国最高法院案件中推翻关系的能力，揭示了模型在时间敏感性、推理深度和上下文依赖性方面的局限性，并提出了一种更贴近真实法律推理任务的长上下文基准。
摘要：
具有扩展上下文窗口的大型语言模型（LLMs）在复杂的法律推理任务中展现出潜力，但它们对长法律文档的理解能力仍缺乏充分评估。开发能够捕捉真实、高风险任务的长上下文基准仍是该领域的主要挑战，因为大多数现有评估依赖于简化的人工合成任务，无法代表真实世界文档理解的复杂性。推翻关系是普通法教义中的基础要素，常见于司法意见中。它们为长文档法律理解提供了聚焦且重要的测试环境，与法律专业人士实际操作高度相似。我们对最先进的LLMs在识别美国最高法院案件中的推翻关系进行了评估，使用了236对案件的数据集。我们的评估揭示了三个关键限制：（1）时代敏感性——模型在历史案件上的表现比现代案件差，揭示了其训练中的根本时间偏见；（2）浅层推理——模型依赖于浅层的逻辑启发式方法而非深入的法律理解；（3）上下文依赖的推理失败——在复杂的开放式任务中，模型会产生时间上不可能的关系，尽管在简单的上下文中能保持基本的时间意识。我们的工作贡献了一个基准，填补了现实长上下文评估中的关键空白，提供了一个与实际法律推理任务的复杂性和风险相匹配的环境。
英文摘要：
Large language models (LLMs) with extended context windows show promise for complex legal reasoning tasks, yet their ability to understand long legal documents remains insufficiently evaluated. Developing long-context benchmarks that capture realistic, high-stakes tasks remains a significant challenge in the field, as most existing evaluations rely on simplified synthetic tasks that fail to represent the complexity of real-world document understanding. Overruling relationships are foundational to common-law doctrine and commonly found in judicial opinions. They provide a focused and important testbed for long-document legal understanding that closely resembles what legal professionals actually do. We present an assessment of state-of-the-art LLMs on identifying overruling relationships from U.S. Supreme Court cases using a dataset of 236 case pairs. Our evaluation reveals three critical limitations: (1) era sensitivity -- the models show degraded performance on historical cases compared to modern ones, revealing fundamental temporal bias in their training; (2) shallow reasoning -- models rely on shallow logical heuristics rather than deep legal comprehension; and (3) context-dependent reasoning failures -- models produce temporally impossible relationships in complex open-ended tasks despite maintaining basic temporal awareness in simple contexts. Our work contributes a benchmark that addresses the critical gap in realistic long-context evaluation, providing an environment that mirrors the complexity and stakes of actual legal reasoning tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
