法律推理不是律师工作:重新思考面向自助诉讼的司法基准
Legal Reasoning Is Not Lawyering: Rethinking Legal Benchmarks for Pro Se Access to Justice
AI总结:
本文指出当前法律AI基准仅评估专家预处理后的输入,忽略了自助诉讼者提供的杂乱、有误的提示,导致模型性能高估;通过实验展示上下界差距,呼吁建立直接衡量鲁棒性的基准。
摘要:
法律AI基准研究经常假设大型语言模型可以改善司法可及性,包括帮助那些无法聘请律师来理解和行使法律权利的人。我们认为,当前的基准不足以支持这一假设,因为它们评估的是已经由法律专家预处理过的输入上的法律推理,这衡量的是模型性能的上限。司法可及性取决于下限:当输入来自自助诉讼者时,模型的表现如何,这些提示可能包含杂乱叙述、隐藏事实、遗漏、民间法律假设和表面错误。这些退化与一般机器学习文献中已知导致LLM性能下降的条件相当,包括长上下文敏感性、欠指定、幻觉和拼写扰动。我们将自助诉讼文献中的证据与这一机器学习研究体系联系起来,并在法律基准LEXam上进行了小规模扰动实验,以说明这两个界限之间的差距。如果模型开发继续专注于仅衡量上限的基准,这一差距可能仍然隐藏甚至扩大。最后,我们呼吁建立直接衡量在类似自助诉讼输入下鲁棒性的法律基准,以便关于法律AI改善司法可及性的主张能够变得可经验验证。
英文摘要:
Legal AI benchmark research frequently invokes the assumption that large language models can improve access to justice, including for people who cannot access lawyers in order to understand and exercise their legal rights. We argue that current benchmarks are not equipped to support this assumption because they evaluate legal reasoning over inputs that have already been preprocessed by legal experts, which measures the upper bound of model performance. Access to justice depends on a lower bound: how models perform when inputs come from pro se litigants, whose prompts may contain noisy narratives, buried facts, omissions, folk-legal assumptions, and surface-level errors. These degradations are comparable to conditions under which LLMs are known to degrade in the general machine learning literature, including long-context sensitivity, underspecification, hallucination, and typographical perturbations. We connect evidence from pro se literature with this body of machine learning research and present a small perturbation experiment on LEXam, a legal benchmark, to illustrate the gap between these two bounds. If model development continues to focus on benchmarks that measure only the upper bound, this gap may remain hidden or even widen. We conclude by calling for legal benchmarks that directly measure robustness under pro se-like inputs so that access-to-justice claims about legal AI can become empirically testable.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。