大语言模型会编造法律引用吗？关于沙特数据保护法和通用数据保护条例的双语基准测试
Do LLMs Fabricate Legal Citations? A Bilingual Benchmark on Saudi Data Protection Law and the GDPR
- IT Department, King Abdulaziz University, Jeddah, Saudi Arabia（国王阿卜杜勒阿齐兹大学信息科技系）
机构由 AI 辅助整理，请以论文原文为准。
中文总结 AI 辅助
研究大语言模型对法律引用的编造情况，通过120个双语问题的基准测试评估三个模型，发现对GDPR和沙特PDPL存在管辖权差距，编造与法律管辖权有关，模型置信度无法保障，提出用逐字验证保障合规筛选。
中文摘要 AI 辅助
组织和监管机构越来越多地向大语言模型咨询合规问题，但错误的法规引用可能会悄悄进入法律建议、合规文件和政策决策中。我们引入了一个包含120个问题的双语基准测试，探究免费可用的大语言模型是否会为欧盟通用数据保护条例（GDPR）和沙特个人数据保护法（PDPL）编造条款引用。该基准测试将直接引用检索问题与错误前提验证探针以及故意无法回答的“陷阱”问题配对。每个问题都用阿拉伯语和英语提出，所有评分均针对人工验证的黄金参考进行全自动评分。评估三个免费可用的模型后发现存在巨大的管辖权差距：GDPR的引用准确率接近上限，而沙特PDPL的引用大多是编造的，且编造率与查询语言无关；最高编造率源于法规与条例的混淆，91%的编造引用置信度≥0.8。编造情况与法律管辖权有关，而非查询语言，模型置信度也无法提供保护，这表明必须通过逐字验证保障措施，而非模型自身置信度，来控制对大语言模型进行合规筛选的机构依赖。
英文摘要
Organizations and regulators increasingly consult large language models (LLMs) for regulatory-compliance questions, yet a wrong statutory citation can silently propagate into legal advice, compliance documentation, and policy decisions. We introduce a bilingual benchmark of 120 questions probing whether freely accessible LLMs fabricate article citations for two data-protection instruments: the EU General Data Protection Regulation (GDPR) and the Saudi Personal Data Protection Law (PDPL). The benchmark pairs direct citation retrieval questions with false premise verification probes and deliberately unanswerable "trap" questions -- including questions about a repealed article and about deadlines that exist only in implementing regulations, not in the law itself. Every question is posed in both Arabic and English, and all scoring is fully automatic against a manually verified gold reference. Evaluating three freely accessible models (Gemini 2.5 Flash, GPT-OSS-120B, Nemotron-3-Super-120B), we find a dramatic jurisdiction gap: near-ceiling citation accuracy on the GDPR (94-100% on direct retrieval) against majority fabrication on the Saudi PDPL (60-77%), invariant to query language; the highest fabrication rates (67%) arise from statute-vs-regulations confusion, and 91% of fabricated citations are asserted with confidence >= 0.8. Fabrication tracks the jurisdiction of the law, not the language of the query, and model confidence provides no protection -- indicating that verbatim-verification safeguards, rather than model self confidence, must gate any institutional reliance on LLMs for compliance screening.
