arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.16470cs.CLcs.AI
收藏

面向低资源与高资源语言的大语言模型机器翻译幻觉检测

Machine Translation Hallucination Detection for Low and High Resource Languages using Large Language Models

University College London(伦敦大学学院);Mila - Quebec AI Institute(米拉-魁北克人工智能研究所);McGill University(麦吉尔大学);Meta

Kenza Benkirane, Laura Gongas, Shahar Pelles, Naomi Fuchs, Joshua Darmon, Pontus Stenetorp, David Ifeoluwa Adelani, Eduardo Sánchez

AI总结:

该研究针对机器翻译幻觉检测在低资源语言上的局限,评估结合大语言模型与多语言嵌入语义相似度的句子级检测方法,发现不同模型在高低资源语言上表现有差异,验证了大语言模型的检测潜力。

摘要:

大规模多语言机器翻译系统的最新进展显著提升了翻译准确率,但即便是性能最优的系统仍会生成幻觉内容,严重影响用户信任。机器翻译(MT)中的幻觉检测仍是一项关键挑战,尤其是现有方法在高资源语言(HRLs)上表现出色,但应用于低资源语言(LRLs)时存在明显局限。本文评估了结合大语言模型(LLMs)与大规模多语言嵌入语义相似度的句子级幻觉检测方法。研究覆盖16个语言方向,涉及高资源语言、低资源语言,包含多种文字系统。我们发现模型选择对性能至关重要:平均而言,对于高资源语言,Llama3-70B比此前的最优方法性能高出最多0.16 MCC(马修斯相关系数);但对于低资源语言,Claude Sonnet的平均表现优于其他大语言模型,MCC高出0.03。本研究的核心结论是,尽管大语言模型未经过任何机器翻译任务的专门训练,其性能仍可与此前提出的模型相当甚至更优,但在低资源语言上的优势相对较弱。

英文摘要:

Recent advancements in massively multilingual machine translation systems have significantly enhanced translation accuracy; however, even the best performing systems still generate hallucinations, severely impacting user trust. Detecting hallucinations in Machine Translation (MT) remains a critical challenge, particularly since existing methods excel with High-Resource Languages (HRLs) but exhibit substantial limitations when applied to Low-Resource Languages (LRLs). This paper evaluates sentence-level hallucination detection approaches using Large Language Models (LLMs) and semantic similarity within massively multilingual embeddings. Our study spans 16 language directions, covering HRLs, LRLs, with diverse scripts. We find that the choice of model is essential for performance. On average, for HRLs, Llama3-70B outperforms the previous state of the art by as much as 0.16 MCC (Matthews Correlation Coefficient). However, for LRLs we observe that Claude Sonnet outperforms other LLMs on average by 0.03 MCC. The key takeaway from our study is that LLMs can achieve performance comparable or even better than previously proposed models, despite not being explicitly trained for any machine translation task. However, their advantage is less significant for LRLs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑