Gaps or Hallucinations? Gazing into Machine-Generated Legal Analysis for Fine-grained Text Evaluations
Johns Hopkins University(约翰斯·霍普金斯大学);Télécom Paris(巴黎电信学院);Institut Polytechnique de Paris(巴黎综合理工学院);University of Maryland, Carey School of Law(马里兰大学凯里法学院)
Abe Bohan Hou, William Jurayj, Nils Holzenberger, Andrew Blair-Stanek, Benjamin Van Durme
Large Language Models (LLMs) show promise as a writing aid for professionals performing legal analyses. However, LLMs can often hallucinate in this setting, in ways difficult to recognize by non-professionals and existing text evaluation metrics. In this work, we pose the question: when can machine-generated legal analysis be evaluated as acceptable? We introduce the neutral notion of gaps, as opposed to hallucinations in a strict erroneous sense, to refer to the difference between human-written and machine-generated legal analysis. Gaps do not always equate to invalid generation. Working with legal experts, we consider the CLERC generation task proposed in Hou et al. (2024b), leading to a taxonomy, a fine-grained detector for predicting gap categories, and an annotated dataset for automatic evaluation. Our best detector achieves 67% F1 score and 80% precision on the test set. Employing this detector as an automated metric on legal analysis generated by SOTA LLMs, we find around 80% contain hallucinations of different kinds.
Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models
Yale University(耶鲁大学);Yale Law School(耶鲁法学院);Stanford University(斯坦福大学);Stanford Law School(斯坦福法学院);Stanford Institute for Economic Policy Research(斯坦福经济政策研究所);Regulation, Evaluation, and Governance Lab(规制、评估与治理实验室)
Place Matters: Comparing LLM Hallucination Rates for Place-Based Legal Queries
School of Computing and Information Systems, The University of Melbourne, Australia(计算与信息系统学院,墨尔本大学,澳大利亚);The Centre for Artificial Intelligence and Digital Ethics(人工智能与数字伦理中心);Melbourne Law School, The University of Melbourne, Australia(墨尔本法学院,墨尔本大学,澳大利亚)
From Judgments to Issues: Structured Extraction of Legal Reasoning with Citation-Hallucination Control
Quantitative and Digital Law Laboratory(量化与数字法律实验室);CIRSFID - Alma AI, Faculty of Law University of Bologna(CIRSFID - 阿尔玛AI,博洛尼亚大学法学院);Robert Schuman Centre European University Institute(罗伯特·施曼中心欧洲大学研究所)
CogniBench: A Legal-inspired Framework and Dataset for Assessing Cognitive Faithfulness of Large Language Models
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Hunyuan AI Digital Human, Tencent(文心一言数字人,腾讯);Beijing University of Posts and Telecommunications(北京邮电大学);Shanghai AI Laboratory(上海人工智能实验室)
DeliLaw: A Chinese Legal Counselling System Based on a Large Language Model
University of Chinese Academy of Sciences(中国科学院大学);Shenzhen Institute of Advanced Technology, CAS(中国科学院深圳先进技术研究院);Hebei University(河北大学);University of Science and Technology of China(中国科学技术大学);South University of Science and Technology of China(南方科技大学)
Evaluating Test-Time Scaling LLMs for Legal Reasoning: OpenAI o1, DeepSeek-R1, and Beyond
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Guanghua Law School, Zhejiang University(浙江大学光华法学院);School of Software Technology, Zhejiang University(浙江大学软件学院);Law & AI Lab, Zhejiang University(浙江大学法律与人工智能实验室);Shanghai AI Laboratory(上海人工智能实验室)
Legal Rule Induction: Towards Generalizable Principle Discovery from Analogous Judicial Precedents
Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China(香港科技大学计算机科学与工程系);School of Law, Tsinghua University, Beijing, China(清华大学法学院)
Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Models
Beijing University of Posts and Telecommunications(北京邮电大学);Shihezi University(石河子大学);Beihang University(北航);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Jilin University(吉林大学);Central South University(中南大学);Jinan University(暨南大学)
DWTSumm: Discrete Wavelet Transform for Document Summarization
School of Engineering and Applied Science, George Washington University(乔治华盛顿大学工程与应用科学学院);Faculty of Computers and Artificial Intelligence, Cairo University(开罗大学计算机与人工智能学院);Language Technologies Institute, Carnegie Mellon University(卡内基梅隆大学语言技术研究所)
Retrieval Augmented Generation-based Large Language Models for Bridging Transportation Cybersecurity Legal Knowledge Gaps
The University of Texas at Dallas(德克萨斯大学达拉斯分校);Alabama Transportation Institute, The University of Alabama(阿拉巴马大学阿拉巴马交通研究所);The University of Alabama(阿拉巴马大学);Clemson University(克莱姆森大学)
SelfCheck-Eval: A Multi-Module Framework for Zero-Resource Hallucination Detection in Large Language Models
Department of Mathematics and Computer Science, University of Catania(卡塔尼亚大学数学与计算机科学系);L3S Research Center, Leibniz University Hannover(汉诺威莱布尼茨大学L3S研究中心)
When Large Language Models Meet Law: Dual-Lens Taxonomy, Technical Advances, and Ethical Governance
School of Law, China University of Political Science and Law(中国政法大学法学院);Zhejiang University of Finance and Economics Dongfang College(浙江财经大学东洋学院);School of Information Management for Law, China University of Political Science and Law(中国政法大学信息管理法学院);Department of Artificial Intelligence, Chung-Ang University(成均馆大学人工智能系);Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University(香港理工大学数据科学与人工智能系)