Neural sequence models can generate highly fluent sentences, but recent studies have also shown that they are also prone to hallucinate additional content not supported by the input. These variety of fluent but wrong outputs are particularly problematic, as it will not be possible for users to tell they are being presented incorrect content. To detect these errors, we propose a task to predict whether each token in the output sequence is hallucinated (not contained in the input) and collect new manually annotated evaluation sets for this task. We also introduce a method for learning to detect hallucinations using pretrained language models fine tuned on synthetic data that includes automatically inserted hallucinations Experiments on machine translation (MT) and abstractive summarization demonstrate that our proposed approach consistently outperforms strong baselines on all benchmark datasets. We further demonstrate how to use the token-level hallucination labels to define a fine-grained loss over the target sequence in low-resource MT and achieve significant improvements over strong baseline methods. We also apply our method to word-level quality estimation for MT and show its effectiveness in both supervised and unsupervised settings. Codes and data available at https://github.com/violet-zct/fairseq-detect-hallucination.
NCL-UoR at SemEval-2025 Task 3: Detecting Multilingual Hallucination and Related Observable Overgeneration Text Spans with Modified RefChecker and Modified SeflCheckGPT
School of Computing, Newcastle University（新萨勒姆大学计算机学院）；Department of Computer Science, University of Reading（阅读大学计算机科学系）；Previously at School of Computing, Newcastle University（新萨勒姆大学计算机学院）
HalluScore: Large Language Model Hallucination Question Answering Benchmark
Department of Information and Computer Science, King Fahd University of Petroleum and Minerals（国王法赫德石油与矿物大学信息与计算机科学系）；SDAIA-KFUPM Joint Research Center for Artificial Intelligence（SDAIA-KFUPM人工智能联合研究中心）
Detecting Contextual Hallucinations in LLMs with Frequency-Aware Attention
Department of Informatics, King's College London, UK（伦敦国王学院信息学院）；Department of Statistics, University of Warwick, UK（沃里克大学统计系）；School of Computer Science, The University of Sheffield, UK（谢菲尔德大学计算机科学学院）；The Alan Turing Institute, UK（艾伦·图灵研究所）
Enhanced Hallucination Detection in Neural Machine Translation through Simple Detector Aggregation
CentraleSupélec（中央高等电力学院）；Universite Paris-Saclay（巴黎萨克雷大学）；Inria（法国国家信息与自动化研究所）；CEA（法国原子能和替代能源委员会）；digeiz；Equall；Instituto de Telecomunicações（电信研究所）；University of Lisbon（里斯本大学）；Unbabel
Imitate Before Detect: Aligning Machine Stylistic Preference for Machine-Revised Text Detection
Fudan University（复旦大学）；South China University of Technology（华南理工大学）；Wuhan University（武汉大学）；Fenz AI；UC San Diego（加州大学圣地亚哥分校）；UIUC（伊利诺伊大学厄巴纳-香槟分校）；CMU（卡内基梅隆大学）；PolyU（香港理工大学）；Stanford University（斯坦福大学）；NUS (Chongqing) Research Institute（新加坡国立大学（重庆）研究院）；Georgia Tech（佐治亚理工学院）
