检索增强生成（RAG）为法律AI提供了显著潜力，但系统性的基准测试却十分稀少。先前的工作引入了LaborBench，用于基于美国劳工部（DOL）律师经过几个月的详尽手动枚举所有美国州失业保险要求的显性真实情况，来评估RAG模型的表现。先前的工作发现标准RAG在布尔任务上的表现很差（准确率为70%）。在此，我们评估了三个新兴工具，这些工具此前未在LaborBench上进行评估：Statutory Research Assistant（STARA），一个定制的立法研究工具，以及由Westlaw和LexisNexis推出的两个商业工具，它们展示了营销AI立法调研能力。我们做出了五个主要贡献。首先，我们展示了STARA实现了显著的性能提升，将准确率提升到83%。其次，我们发现商业平台表现不佳，准确率分别为58%（Westlaw AI）和64%（Lexis+ AI），甚至比标准RAG更差。第三，我们进行了全面的错误分析，将我们的输出与DOL律师编写的输出进行比较，并记录了推理错误，如混淆相关法律概念和误解立法例外情况，以及检索失败，即相关立法条款未被捕捉。第四，我们发现许多明显的错误实际上是DOL律师自身的重大遗漏，因此STARA的实际准确率是92%。第五，我们通过具体的指导原则为法律RAG指明了前进的道路，为构建能够准确进行多司法管辖区法律研究的AI系统提供可操作的指导。
英文摘要：
Retrieval-augmented generation (RAG) offers significant potential for legal AI, yet systematic benchmarks are sparse. Prior work introduced LaborBench to benchmark RAG models based on ostensible ground truth from an exhaustive, multi-month, manual enumeration of all U.S. state unemployment insurance requirements by U.S. Department of Labor (DOL) attorneys. That prior work found poor performance of standard RAG (70% accuracy on Boolean tasks). Here, we assess three emerging tools not previously evaluated on LaborBench: the Statutory Research Assistant (STARA), a custom statutory research tool, and two commercial tools by Westlaw and LexisNexis marketing AI statutory survey capabilities. We make five main contributions. First, we show that STARA achieves substantial performance gains, boosting accuracy to 83%. Second, we show that commercial platforms fare poorly, with accuracy of 58% (Westlaw AI) and 64% (Lexis+ AI), even worse than standard RAG. Third, we conduct a comprehensive error analysis, comparing our outputs to those compiled by DOL attorneys, and document both reasoning errors, such as confusion between related legal concepts and misinterpretation of statutory exceptions, and retrieval failures, where relevant statutory provisions are not captured. Fourth, we discover that many apparent errors are actually significant omissions by DOL attorneys themselves, such that STARA's actual accuracy is 92%. Fifth, we chart the path forward for legal RAG through concrete design principles, offering actionable guidance for building AI systems capable of accurate multi-jurisdictional legal research.
Decompose-and-Refine: Structured Legal Question Answering with Parametric Retrieval
Graduate School of Artificial Intelligence, POSTECH, Republic of Korea（延世大学人工智能研究生院，韩国POSTECH）；Department of Computer Science and Engineering, POSTECH, Republic of Korea（POSTECH计算机科学与工程系，韩国POSTECH）
Johns Hopkins University（约翰斯·霍普金斯大学）；U. of Maryland Carey School of Law（马里兰大学凯里法学院）
Legislation can be viewed as a body of prescriptive rules expressed in natural language. The application of legislation to facts of a case we refer to as statutory reasoning, where those facts are also expressed in natural language. Computational statutory reasoning is distinct from most existing work in machine reading, in that much of the information needed for deciding a case is declared exactly once (a law), while the information needed in much of machine reading tends to be learned through distributional language statistics. To investigate the performance of natural language understanding…
Statutory article retrieval (SAR), the task of retrieving statute law articles relevant to a legal question, is a promising application of legal text processing. In particular, high-quality SAR systems can improve the work efficiency of legal professionals and provide basic legal assistance to citizens in need at no cost. Unlike traditional ad-hoc information retrieval, where each document is considered a complete source of information, SAR deals with texts whose full sense depends on complementary information from the topological organization of statute law. While existing works ignore these …
LabourCrew: A Multi-Agent RAG Framework for Trustworthy Adversarial Deliberation and Statutory Reasoning over Labour Law
Ahsanullah University of Science and Technology（阿赫萨努拉科技大学）；Jashore University of Science and Technology（杰肖尔科技大学）；American International University - Bangladesh（美国国际大学-孟加拉）
GreekBarRetrieval: A Benchmark for Greek Statutory Retrieval
Archimedes, Athena Research Center（阿基米德雅典娜研究中心）；Institute for Language and Speech Processing, Athena Research Center（雅典娜研究中心语言与语音处理研究所）；Athens University of Economics Business（雅典经济与商业大学）
LegalRAG: A Hybrid RAG System for Multilingual Legal Information Retrieval
Apurba-NSU R&D Lab, Department of Electrical and Computer Engineering, North South University, Dhaka, Bangladesh（北南大学电气与计算机工程系，北南大学阿普尔巴-NSU研发实验室，达卡，孟加拉国）；Apurba Technologies, Sunnyvale, CA 94085, USA（阿普尔巴技术公司，硅谷，加利福尼亚州，美国）；Fordham University, New York, USA（福丁大学，纽约，美国）
Statutory article retrieval is the task of automatically retrieving law articles relevant to a legal question. While recent advances in natural language processing have sparked considerable interest in many legal tasks, statutory article retrieval remains primarily untouched due to the scarcity of large-scale and high-quality annotated datasets. To address this bottleneck, we introduce the Belgian Statutory Article Retrieval Dataset (BSARD), which consists of 1,100+ French native legal questions labeled by experienced jurists with relevant articles from a corpus of 22,600+ Belgian law articles…
Towards Reliable Retrieval in RAG Systems for Large Legal Datasets
Technical University of Darmstadt（德意志联邦共和国达姆斯塔特技术大学）；University of Trento（特伦托大学）；University of Florence（佛罗伦萨大学）；University of Bologna（博洛尼亚大学）；European University Institute（欧洲大学研究所）
CoAL-RAG: A Complexity-Aware Legal Retrieval-Augmented Generation Method
North China University of Technology（北方工业大学）；Beijing Key Laboratory of Key Technologies for AI+ Domain Applications（北京市AI+领域应用关键技术重点实验室）；Beijing Key Laboratory on Integration and Analysis of Large-scale Stream Data（大规模流数据集成与分析技术北京市重点实验室）
From Query to Counsel: Structured Reasoning with a Multi-Agent Framework and Dataset for Legal Consultation
Australian Artificial Intelligence Institute (AAII), University of Technology Sydney（悉尼科技大学澳大利亚人工智能研究所）；School of Computer Science, University of Birmingham（伯明翰大学计算机科学学院）
NyayaRAG: Realistic Legal Judgment Prediction with RAG under the Indian Common Law System
Indian Institute of Technology Kanpur（印度理工学院坎浦尔分校）；SRM Institute of Science and Technology（SRM科学与技术学院）；Symbiosis Law School Pune（苏比斯法学院浦那分校）；IISER Kolkata（印度科学研究所库勒卡布尔分校）；University of Birmingham, Dubai, United Arab Emirates（布里斯托尔大学迪拜分校）
CanLegalRAGBench: Evaluating Retrieval-Augmented Generation on Canadian Case Law
Department of Computer Science, University of British Columbia（不列颠哥伦比亚大学计算机科学系）；Vector Institute（向量研究所）；CIFAR AI Chair Peter A. Allard School of Law, University of British Columbia（不列颠哥伦比亚大学彼得·A·艾尔德法学院）
PAKTON: A Multi-Agent Framework for Question Answering in Long Legal Agreements
School of Electrical and Computer Engineering, AILS Laboratory, National Technical University of Athens（电气与计算机工程学院，AILS实验室，国家技术大学雅典）；Instituto de Telecomunicações（电信研究所）
