arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.00340cs.AI
收藏

更好的CLAUSE:用于审计LLM法律推理能力的差异基准

Better Call CLAUSE: A Discrepancy Benchmark for Auditing LLMs Legal Reasoning Capabilities

Arizona State University(亚利桑那州立大学)

Manan Roy Choudhury, Adithya Chandramouli, Mannan Anand, Vivek Gupta

AI总结:

CLAUSE是一个用于评估LLM法律推理能力的基准,通过生成现实扰动合同检测细微法律错误,揭示LLM在识别和解释法律缺陷方面的不足。

摘要:

大语言模型(LLM)在高风险法律工作中迅速整合,暴露了一个关键缺口:目前没有基准可以系统地检验其可靠性,以应对现实合同中存在的细微、对抗性且往往微妙的缺陷。为了解决这一问题,我们引入了CLAUSE,这是一个首创的基准,旨在评估LLM法律推理的脆弱性。我们研究LLM检测和推理细微差异的能力,通过从CUAD和ContractNLI等基础数据集中生成超过7500个现实扰动合同。我们的新型基于角色的管道生成10种不同的异常类别,然后通过检索增强生成(RAG)系统与官方法规进行验证以确保法律准确性。我们使用CLAUSE评估领先LLM检测嵌入法律缺陷并解释其重要性的能力。我们的分析显示一个关键弱点:这些模型常常忽略细微错误,甚至更难以在法律上证明这些错误。我们的工作为识别和纠正法律AI中的推理失败指明了一条道路。

英文摘要:

The rapid integration of large language models (LLMs) into high-stakes legal work has exposed a critical gap: no benchmark exists to systematically stress-test their reliability against the nuanced, adversarial, and often subtle flaws present in real-world contracts. To address this, we introduce CLAUSE, a first-of-its-kind benchmark designed to evaluate the fragility of an LLM's legal reasoning. We study the capabilities of LLMs to detect and reason about fine-grained discrepancies by producing over 7500 real-world perturbed contracts from foundational datasets like CUAD and ContractNLI. Our novel, persona-driven pipeline generates 10 distinct anomaly categories, which are then validated against official statutes using a Retrieval-Augmented Generation (RAG) system to ensure legal fidelity. We use CLAUSE to evaluate leading LLMs' ability to detect embedded legal flaws and explain their significance. Our analysis shows a key weakness: these models often miss subtle errors and struggle even more to justify them legally. Our work outlines a path to identify and correct such reasoning failures in legal AI.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑