Hallucinations as Orthogonal Noise: Inference-Time Manifold Alignment via Dynamic Contextual Orthogonalization
Xi’an Jiaotong University(西安交通大学);Xingchen AGI Lab(星辰AGI实验室);China Telecom AI Technology (Beijing) Co., Ltd.(中国电信人工智能技术(北京)有限公司);Institute of Artificial Intelligence, China Telecom(中国电信人工智能研究院);University of Science and Technology of China(中国科学技术大学);Tsinghua University(清华大学)
Hallucination in Large Language Models (LLMs), characterized by the generation of content inconsistent with contextual facts or logical constraints -- remains a persistent challenge for reliable deployment. In this work, we address this issue through a geometric framework rooted in the linear representation hypothesis. We propose that hallucinations manifest as orthogonal noise relative to the semantic manifold of the residual stream. Specifically, we hypothesize that while attention heads ideally propagate information congruent with the context subspace, hallucinations arise when specific heads introduce components orthogonal to this subspace, disrupting the coherence of the latent representation. Based on this formulation, we introduce Dynamic Contextual Orthogonalization (DCO), an inference-time intervention method. DCO utilizes the input residual stream as a dynamic context anchor to perform orthogonal decomposition on attention head outputs. To distinguish between context-aligned semantic updates and divergent noise, DCO employs a layer-wise Z-score suppression mechanism that selectively attenuates outlier orthogonal components based on statistical distributions. Evaluations on Llama-3-8B and 70B across benchmarks such as XSum, NQ-Swap, and IFEval demonstrate that DCO achieves superior contextual faithfulness compared to state-of-the-art intervention baselines. Furthermore, DCO maintains high performance on knowledge-intensive tasks like TriviaQA and TruthfulQA, effectively mitigating the trade-off between hallucination suppression and parametric knowledge retention often observed in existing methods. Our findings validate the geometric interpretation of hallucinations and establish DCO as a computationally efficient approach for enforcing manifold alignment.Our code is available at https://github.com/Harry-Miral/DCO
Zero-resource Hallucination Detection for Text Generation via Graph-based Contextual Knowledge Triples Modeling
College of Computer, National University of Defense Technology(国防科技大学计算机学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院)
Copy-Paste to Mitigate Large Language Model Hallucinations
Department of Computer Science, Tianjin University of Technology(天津理工大学计算机学院);National Institute of Health Data Science, Peking University(北京大学国家健康数据科学研究院);Tencent Jarvis Lab(腾讯Jarvis实验室)
Detecting Contextual Hallucinations in LLMs with Frequency-Aware Attention
Department of Informatics, King's College London, UK(伦敦国王学院信息学院);Department of Statistics, University of Warwick, UK(沃里克大学统计系);School of Computer Science, The University of Sheffield, UK(谢菲尔德大学计算机科学学院);The Alan Turing Institute, UK(艾伦·图灵研究所)
Catch Me if You Search: When Contextual Web Search Results Affect the Detection of Hallucinations
The Pennsylvania State University(宾夕法尼亚州立大学);Department of Communication & Center for Trustworthy Artificial Intelligence (CTAI)(通信系与可信人工智能中心);Seoul National University(首尔国立大学)
DeCoRe: Decoding by Contrasting Retrieval Heads to Mitigate Hallucinations
University of Edinburgh(爱丁堡大学);Centre for AI, Data Science & Artificial Intelligence, R&D, AstraZeneca(阿斯利康研发中心人工智能、数据科学与人工智能中心);University College London(伦敦大学学院)
An Evolutionary Large Language Model for Hallucination Mitigation
BIOSTIM Laboratory Faculty of Medicine Salah Boubnider University Constantine, Algeria(BIOSTIM实验室医学院萨拉赫·布宾德大学阿尔及利亚科纳克特);Department of IFA Faculty of NTIC Abdelhamid Mehri University Constantine, Algeria(IFA部门NTIC学院阿卜杜勒哈米德·梅赫里大学阿尔及利亚科纳克特)
Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Science(中国科学院自动化研究所);Hong Kong University of Science and Technology(香港科技大学)
A Scoping Review of Natural Language Processing in Addressing Medically Inaccurate Information: Errors, Misinformation, and Hallucination
Biomedical Informatics and Medical Education(生物医学信息学与医学教育);Health AI(健康人工智能);Department of Information Sciences and Technology(信息科学与技术系);Department of Linguistics(语言学系)
MALM: A Multi-Information Adapter for Large Language Models to Mitigate Hallucination
Beijing Institute of Technology(北京理工大学);The Open University(开放大学);Beijing Information Science and Technology University(北京信息科技大学);The Hong Kong Polytechnic University(香港理工大学)
Do Benchmarks Underestimate LLM Performance? Evaluating Hallucination Detection With LLM-First Human-Adjudicated Assessment
Department of Computer Engineering, Hacettepe University(哈切塔佩大学计算机工程系);Department of Computer Engineering, Ankara University(安卡拉大学计算机工程系);Zephlen AI and Information Technologies Inc.(泽夫伦人工智能与信息技术公司)