地点至关重要：比较基于地点的法律查询中大语言模型的幻觉率
Place Matters: Comparing LLM Hallucination Rates for Place-Based Legal Queries
AI总结：
本文提出基于功能主义的比较方法，利用Reddit法律咨询场景构建数据集，评估闭源大语言模型在洛杉矶、伦敦和悉尼的法律信息幻觉率，发现幻觉率与地点显著相关，并揭示幻觉率与多数回答频率负相关，可用于度量模型法律事实预测的不确定性。
摘要：
我们如何有意义地比较大语言模型对一个地方与另一个地方的法律知识？量化这些差异对于理解基于大语言模型的聊天机器人用户所获得的法律信息质量是否因其所在地点而异至关重要。然而，获得有意义的比较指标具有挑战性，因为不同地方的法律制度本身并不容易比较。在这项工作中，我们提出了一种基于比较法中的功能主义概念来获取地点间指标的方法。我们构建了一个事实场景数据集，这些场景取自 Reddit 上寻求家庭、住房、就业、犯罪和交通问题法律建议的用户帖子。我们利用这些场景，让大语言模型针对洛杉矶、伦敦和悉尼的每个场景生成相关法律的摘要。这些摘要通常涉及某项立法条款，并由人工评估其是否存在幻觉。我们表明，领先的闭源大语言模型对法律信息的幻觉率与地点显著相关。这表明这些模型提供的法律解决方案的质量在地理分布上并不均匀。此外，我们展示了当对大语言模型进行多次采样时，幻觉率与多数回答的出现频率之间存在强烈的负相关关系，这为模型对法律事实预测的不确定性提供了一种度量。
英文摘要：
How do we make a meaningful comparison of a large language model's knowledge of the law in one place compared to another? Quantifying these differences is critical to understanding if the quality of the legal information obtained by users of LLM-based chatbots varies depending on their location. However, obtaining meaningful comparative metrics is challenging because legal institutions in different places are not themselves easily comparable. In this work we propose a methodology to obtain place-to-place metrics based on the comparative law concept of functionalism. We construct a dataset of factual scenarios drawn from Reddit posts by users seeking legal advice for family, housing, employment, crime and traffic issues. We use these to elicit a summary of a law from the LLM relevant to each scenario in Los Angeles, London and Sydney. These summaries, typically of a legislative provision, are manually evaluated for hallucinations. We show that the rate of hallucination of legal information by leading closed-source LLMs is significantly associated with place. This suggests that the quality of legal solutions provided by these models is not evenly distributed across geography. Additionally, we show a strong negative correlation between hallucination rate and the frequency of the majority response when the LLM is sampled multiple times, suggesting a measure of uncertainty of model predictions of legal facts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
