CLaw：大型语言模型中中文法律知识的基准测试——细粒度语料与推理分析
CLaw: Benchmarking Chinese Legal Knowledge in Large Language Models - A Fine-grained Corpus and Reasoning Analysis
AI总结：
针对LLM法律知识可靠性问题，提出CLaw基准，含306部中国法律细粒度语料与254个案例推理实例，发现模型复现法条困难，强调知识检索与推理协同。
摘要：
大型语言模型（LLMs）越来越多地被用于分析法律文本并引用相关法条，然而其可靠性常常因通用预训练而受损，因为预训练虽摄入法律文本但缺乏专门聚焦，掩盖了其法律知识的真实深度。本文介绍了CLaw，一个专门设计用于细致评估LLMs在中文法律知识及其推理应用方面表现的新型基准。CLaw包含两个关键组成部分：（1）一个全面、细粒度的语料库，涵盖全部306部中国国家法律，细分至款级，并纳入精确的历史修订时间点，以进行严格的召回评估（共64,849条）；（2）一组具有挑战性的254个基于案例的推理实例，源自中国最高人民法院精选材料，用于评估法律知识的实际应用。我们的实证评估显示，大多数当代LLMs在忠实复现法律条文方面存在显著困难。由于法律条文的准确检索与引用是法律推理的基础，这一缺陷严重削弱了其回复的可靠性。我们认为，要在LLMs中实现可信赖的法律推理，需要准确的知识检索（可能通过监督微调（SFT）或检索增强生成（RAG）加以增强）与强大的通用推理能力之间的稳健协同。这项工作为推进特定领域LLM推理，尤其是在复杂的法律领域，提供了必要的基准和关键见解。
英文摘要：
Large Language Models (LLMs) are increasingly tasked with analyzing legal texts and citing relevant statutes, yet their reliability is often compromised by general pre-training that ingests legal texts without specialized focus, obscuring the true depth of their legal knowledge. This paper introduces CLaw, a novel benchmark specifically engineered to meticulously evaluate LLMs on Chinese legal knowledge and its application in reasoning. CLaw comprises two key components: (1) a comprehensive, fine-grained corpus of all 306 Chinese national statutes, segmented to the subparagraph level and incorporating precise historical revision timesteps for rigorous recall evaluation (64,849 entries), and (2) a challenging set of 254 case-based reasoning instances derived from China Supreme Court curated materials to assess the practical application of legal knowledge. Our empirical evaluation reveals that most contemporary LLMs significantly struggle to faithfully reproduce legal provisions. As accurate retrieval and citation of legal provisions form the basis of legal reasoning, this deficiency critically undermines the reliability of their responses. We contend that achieving trustworthy legal reasoning in LLMs requires a robust synergy of accurate knowledge retrieval--potentially enhanced through supervised fine-tuning (SFT) or retrieval-augmented generation (RAG)--and strong general reasoning capabilities. This work provides an essential benchmark and critical insights for advancing domain-specific LLM reasoning, particularly within the complex legal sphere.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
