arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.02504cs.CL
收藏

多语言幻觉基准:MultiWikiQHalluA

A multilingual hallucination benchmark: MultiWikiQHalluA

Alexandra Institute(亚历山大研究所)

Freja Thoresen, Dan Saattrup Smart

AI总结:

针对现有幻觉评估多语言覆盖不足问题,基于MultiWikiQA构建306种语言合成幻觉数据集,训练30种欧洲语言的分类器,评测主流大模型幻觉率,发现低资源语言幻觉率更高。

摘要:

多数幻觉评估工作聚焦英语,相关结论能否迁移到低资源语言尚不明确。本文研究忠实性幻觉,即模型生成内容流畅合理但偏离给定输入或存在内部不一致的现象。基于多语言MultiWikiQA数据集,利用LettuceDetect框架为306种语言构建合成幻觉数据集,据此训练得到30种欧洲语言的令牌级幻觉分类器。本文针对英语、丹麦语、德语、冰岛语开展模型幻觉评估,使用上述分类器评测Qwen3-0.6B、Qwen3-14B、Gemma-3-12B-IT、cogito-v1-preview-qwen-32B、cogito-v1-preview-llama-70B的幻觉率。分类器评测结果显示Qwen3-0.6B幻觉率显著更高,最多有60%的回答包含至少一处幻觉,峰值出现在冰岛语场景下;更大参数规模的模型幻觉率整体更低,其中cogito-v1-preview-qwen-32B和cogito-v1-preview-llama-70B在多数语言上表现最优。低资源语言尤其是冰岛语的幻觉率始终处于更高水平。

英文摘要:

Most hallucination evaluations focus on English, leaving it unclear whether findings transfer to lower-resource languages. We investigate faithfulness hallucinations, defined as model-generated content that is fluent and plausible but diverges from the provided input or is internally inconsistent. Leveraging the multilingual MultiWikiQA dataset, we utilize the LettuceDetect framework to create synthetic hallucination datasets for 306 languages, from which we train token-level hallucination classifiers for 30 European languages. In this work, we present evaluations of model hallucinations on a selection of languages: English, Danish, German, and Icelandic. Using these classifiers, we evaluate the hallucination rates for Qwen3-0.6B, Qwen3-14B, Gemma-3-12B-IT, cogito-v1-preview-qwen-32B, and cogito-v1-preview-llama-70B. Our classifiers reveal notably higher hallucination rates for Qwen3-0.6B (up to 60\% of answers containing at least one hallucination, peaking in Icelandic) and generally lower rates for larger models, with cogito-v1-preview-qwen-32B and cogito-v1-preview-llama-70B performing best on most languages. Hallucination rates are consistently higher for lower-resource languages, particularly Icelandic.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑