Recent advancements in massively multilingual machine translation systems have significantly enhanced translation accuracy; however, even the best performing systems still generate hallucinations, severely impacting user trust. Detecting hallucinations in Machine Translation (MT) remains a critical challenge, particularly since existing methods excel with High-Resource Languages (HRLs) but exhibit substantial limitations when applied to Low-Resource Languages (LRLs). This paper evaluates sentence-level hallucination detection approaches using Large Language Models (LLMs) and semantic similarity within massively multilingual embeddings. Our study spans 16 language directions, covering HRLs, LRLs, with diverse scripts. We find that the choice of model is essential for performance. On average, for HRLs, Llama3-70B outperforms the previous state of the art by as much as 0.16 MCC (Matthews Correlation Coefficient). However, for LRLs we observe that Claude Sonnet outperforms other LLMs on average by 0.03 MCC. The key takeaway from our study is that LLMs can achieve performance comparable or even better than previously proposed models, despite not being explicitly trained for any machine translation task. However, their advantage is less significant for LRLs.
Can LLMs Detect Intrinsic Hallucinations in Paraphrasing and Machine Translation?
RISE Research Institutes of Sweden, Department of Computer Science(瑞典RISE研究机构,计算机科学系);Uppsala University, Department of Linguistics and Philology(乌普萨拉大学,语言学与哲学系);KTH Royal Institute of Technology, Division of Software and Computer Systems(皇家理工学院,软件与计算机系统系);University of Edinburgh, School of Informatics(爱丁堡大学,信息学院)
Hallucinations in Large Multilingual Translation Models
Instituto de Telecomunicações(电信研究所);Instituto Superior Técnico, University of Lisbon(里斯本大学高等技术学院);University of Edinburgh(爱丁堡大学);CentraleSupélec, Université Paris-Saclay(巴黎萨克雷大学中央理工-高等电力学院);Unbabel(Unbabel公司)
Do LLM hallucination detectors suffer from low-resource effect?
Indian Institute of Technology Kharagpur(印度理工学院卡拉格普尔分校);Ruhr University Bochum(波鸿鲁尔大学);UAR Research Center for Trustworthy Data Science and Security(UAR可信数据科学与安全研究中心)
Zero-resource Hallucination Detection for Text Generation via Graph-based Contextual Knowledge Triples Modeling
College of Computer, National University of Defense Technology(国防科技大学计算机学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院)
Don't Believe Everything You Read: Enhancing Summarization Interpretability through Automatic Identification of Hallucinations in Large Language Models
Mitigating Multilingual Hallucination in Large Vision-Language Models
School of Computer Science & Technology, Huazhong University of Science and Technology(华中科技大学计算机科学与技术学院);School of Computing Science and Technology, Fudan University(复旦大学计算机科学技术学院);Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);School of Computing Science and Technology, Zhejiang Gongshang University(浙江工商大学计算机科学与技术学院);Department of Computer Science and Engineering, The Chinese University of Hong Kong(香港中文大学计算机科学与工程学系)
SelfCheck-Eval: A Multi-Module Framework for Zero-Resource Hallucination Detection in Large Language Models
Department of Mathematics and Computer Science, University of Catania(卡塔尼亚大学数学与计算机科学系);L3S Research Center, Leibniz University Hannover(汉诺威莱布尼茨大学L3S研究中心)
FFE-Hallu:Hallucinations in Fixed Figurative Expressions:Benchmark of Idioms and Proverbs in the Persian Language
Tehran Institute for Advanced Studies, Khatam University, Iran(德黑兰高级研究 institute,Khatam 大学,伊朗);School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran(电气与计算机工程学院,工程学院,德黑兰大学,德黑兰,伊朗)
A Concise Review of Hallucinations in LLMs and their Mitigation
Computer Engineering K J Somaiya Institute of Technology Mumbai, India;Data Science K J Somaiya Institute of Technology Mumbai, India;Information Technology K J Somaiya Institute of Technology Mumbai, India
AutoHall: Automated Factuality Hallucination Dataset Generation for Large Language Models
AGI Institute, School of Computer Science, Shanghai Jiao Tong University(AGI研究院,计算机科学学院,上海交通大学);Shanghai Key Laboratory of Trusted Data Circulation and Governance in Web3(上海可信数据流通与治理Web3重点实验室);School of Media & Communication, Shanghai Jiao Tong University(媒体与传播学院,上海交通大学)
On the Hallucination in Simultaneous Machine Translation
Institute of Computing and Intelligence, Harbin Institute of Technology(哈尔滨工业大学计算与智能学院);College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部)