大型法律虚构：剖析大型语言模型中的法律幻觉
Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models
AI总结：
本文首次系统研究了大型语言模型在法律领域的幻觉问题，发现ChatGPT 4和Llama 2在联邦法院案件问答中幻觉率分别高达58%和88%，并建立了法律幻觉类型学，警示LLMs在法律任务中快速无监督整合的风险。
摘要：
大型语言模型（LLMs）是否了解法律？这些模型正越来越多地被用于增强法律实践、教育和研究，然而其革命性潜力正受到幻觉——即与法律事实不一致的文本输出——的威胁。我们首次提供了关于这些幻觉的系统性证据，记录了LLMs在不同司法管辖区、法院、时间段和案件中的表现差异。我们的工作做出了四项关键贡献。第一，我们建立了一种法律幻觉的类型学，为该领域的未来研究提供了一个概念框架。第二，我们发现法律幻觉的普遍程度令人震惊，当这些模型被问及关于随机联邦法院案件的具体、可验证的问题时，ChatGPT 4出现幻觉的比例为58%，而Llama 2则高达88%。第三，我们展示了在反事实问题设置中，LLMs往往无法纠正用户不正确的法律假设。第四，我们提供了证据表明，LLMs并非总能预测——或者说并非总能知道——它们何时正在产生法律幻觉。综上所述，我们的研究结果警示人们不要将流行的LLMs快速且无监督地整合到法律任务中。即使是经验丰富的律师也必须对法律幻觉保持警惕，而风险最高的是那些本应从LLMs中获益最多的人——自行诉讼的当事人或无法获得传统法律资源的人。
英文摘要：
Do large language models (LLMs) know the law? These models are increasingly being used to augment legal practice, education, and research, yet their revolutionary potential is threatened by the presence of hallucinations -- textual output that is not consistent with legal facts. We present the first systematic evidence of these hallucinations, documenting LLMs' varying performance across jurisdictions, courts, time periods, and cases. Our work makes four key contributions. First, we develop a typology of legal hallucinations, providing a conceptual framework for future research in this area. Second, we find that legal hallucinations are alarmingly prevalent, occurring between 58% of the time with ChatGPT 4 and 88% with Llama 2, when these models are asked specific, verifiable questions about random federal court cases. Third, we illustrate that LLMs often fail to correct a user's incorrect legal assumptions in a contra-factual question setup. Fourth, we provide evidence that LLMs cannot always predict, or do not always know, when they are producing legal hallucinations. Taken together, our findings caution against the rapid and unsupervised integration of popular LLMs into legal tasks. Even experienced lawyers must remain wary of legal hallucinations, and the risks are highest for those who stand to benefit from LLMs the most -- pro se litigants or those without access to traditional legal resources.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
