LLM vs. Lawyers: Identifying a Subset of Summary Judgments in a Large UK Case Law Dataset
Ahmed Izzidien, Holli Sargeant, Felix Steffek
备注：
36 pages, 13 figures. This work was funded by the Nuffield Foundation grant Access to Justice Through Artificial Intelligence. The views expressed are those of the authors and not necessarily of the Foundation. Visit www.nuffieldfoundation.org. We are grateful to Nicola Mathew for excellent research assistance
开展法律计算研究时，高效识别与特定法律问题相关的法院判决数据集是一项关键却颇具挑战的工作。本研究针对现有大型法律语料库研究中存在的空白——即如何从大量英国法院判决语料中筛选出特定类型案件（本研究中为即决判决案件）展开探索。我们对两种计算方法开展了对比分析：（1）基于传统自然语言处理的方法，利用专家生成的关键词和逻辑运算符进行筛选；（2）创新应用Claude 2大语言模型，基于特定内容提示词对案件进行分类。我们使用包含356011份英国法院判决的Cambridge Law Corpus（剑桥法律语料库）开展实验，结果显示大语言模型的加权F1值达到0.94，而关键词方法仅为0.78。尽管经过迭代优化，基于关键词的搜索逻辑仍无法捕捉法律语言中的细微差别。我们识别并提取了3102份即决判决案件，得以梳理其在英国不同法院的分布情况及时间跨度变化。本论文是运用先进自然语言处理技术解决核心法律研究任务的开创性尝试，证明了这些技术能够弥补系统性缺口、提升法律信息的可获取性。我们公开了提取数据集的相关指标，为后续即决判决相关研究提供支持。
英文摘要：
To undertake computational research of the law, efficiently identifying datasets of court decisions that relate to a specific legal issue is a crucial yet challenging endeavour. This study addresses the gap in the literature working with large legal corpora about how to isolate cases, in our case summary judgments, from a large corpus of UK court decisions. We introduce a comparative analysis of two computational methods: (1) a traditional natural language processing-based approach leveraging expert-generated keywords and logical operators and (2) an innovative application of the Claude 2 large language model to classify cases based on content-specific prompts. We use the Cambridge Law Corpus of 356,011 UK court decisions and determine that the large language model achieves a weighted F1 score of 0.94 versus 0.78 for keywords. Despite iterative refinement, the search logic based on keywords fails to capture nuances in legal language. We identify and extract 3,102 summary judgment cases, enabling us to map their distribution across various UK courts over a temporal span. The paper marks a pioneering step in employing advanced natural language processing to tackle core legal research tasks, demonstrating how these technologies can bridge systemic gaps and enhance the accessibility of legal information. We share the extracted dataset metrics to support further research on summary judgments.
ASP2LJ : An Adversarial Self-Play Laywer Augmented Legal Judgment Framework
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems（认知与复杂系统决策智能重点实验室）；Institute of Automation Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Unisound Beijing China（北京Unisound）
Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond English
HiTZ Center, University of the Basque Country (UPV/EHU)（巴斯克大学（UPV/EHU）HiTZ中心）；University of Alicante (UA)（阿利坎特大学（UA））；GOI Institute, Basque Summer University (UEU)（巴斯克夏季大学（UEU）GOI研究所）
OPOR-Bench: Evaluating Large Language Models on Online Public Opinion Report Generation
Communication University of China（中国传媒大学）；Harbin Institute of Technology（哈尔滨工业大学）；China Academy of Railway Sciences Corporation Limited（中国铁道科学研究院集团有限公司）；School of Engineering, Santa Clara University（圣克拉拉大学工程学院）
Bridge-AI Lab（Bridge-AI实验室）；University of Central Florida（中佛罗里达大学）；Auburn University（奥本大学）；Wichita State University（威奇托州立大学）；Gustavus Adolphus College（古斯塔夫斯阿道夫学院）
Is This Just Fantasy? Language Model Representations Reflect Human Judgments of Event Plausibility
Department of Computer Science（计算机科学系）；Department of Cognitive Science（认知科学系）；Google（谷歌）；DeepMind（深度Mind）；Brown University（布朗大学）；Johns Hopkins University（约翰霍普金斯大学）；Department of Cognitive（认知系）；Department of Computer Science & Psychological Sciences（计算机科学与心理学科学系）
