arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.09947cs.CLcs.CY
收藏

Gaps还是Hallucinations?审视机器生成的法律分析以进行细粒度文本评估

Gaps or Hallucinations? Gazing into Machine-Generated Legal Analysis for Fine-grained Text Evaluations

Johns Hopkins University(约翰斯·霍普金斯大学);Télécom Paris(巴黎电信学院);Institut Polytechnique de Paris(巴黎综合理工学院);University of Maryland, Carey School of Law(马里兰大学凯里法学院)

Abe Bohan Hou, William Jurayj, Nils Holzenberger, Andrew Blair-Stanek, Benjamin Van Durme

AI总结:

本研究针对机器生成法律分析中的幻觉问题,引入中立的gaps概念区分人类与机器文本差异,构建分类体系、细粒度检测器及标注数据集,发现约80%的SOTA LLMs生成内容含幻觉。

摘要:

大型语言模型(LLMs)作为专业人员进行法律分析的写作辅助工具展现出潜力。然而,LLMs在此场景中常会产生幻觉,非专业人员和现有文本评估指标难以识别。本研究提出问题:机器生成的法律分析何时可被评估为可接受?我们引入了中立的gaps概念,与严格错误意义上的hallucinations相对,指代人类撰写与机器生成法律分析之间的差异。Gaps并不总是等同于无效生成。我们与法律专家合作,考察了Hou等人(2024b)提出的CLERC生成任务,由此构建了分类体系、预测gap类别的细粒度检测器及用于自动评估的标注数据集。最佳检测器在测试集上达到67%的F1分数和80%的精确率。将该检测器作为SOTA LLMs生成法律分析的自动化评估指标,我们发现约80%包含不同类型的hallucinations。

英文摘要:

Large Language Models (LLMs) show promise as a writing aid for professionals performing legal analyses. However, LLMs can often hallucinate in this setting, in ways difficult to recognize by non-professionals and existing text evaluation metrics. In this work, we pose the question: when can machine-generated legal analysis be evaluated as acceptable? We introduce the neutral notion of gaps, as opposed to hallucinations in a strict erroneous sense, to refer to the difference between human-written and machine-generated legal analysis. Gaps do not always equate to invalid generation. Working with legal experts, we consider the CLERC generation task proposed in Hou et al. (2024b), leading to a taxonomy, a fine-grained detector for predicting gap categories, and an annotated dataset for automatic evaluation. Our best detector achieves 67% F1 score and 80% precision on the test set. Employing this detector as an automated metric on legal analysis generated by SOTA LLMs, we find around 80% contain hallucinations of different kinds.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑