Improving Contextual Faithfulness of Large Language Models via Retrieval Heads-Induced Optimization
Harbin Institute of Technology(哈尔滨工业大学);Peng Cheng Laboratory(鹏城实验室);Northeastern University(东北大学);The University of Hong Kong(香港大学);iFLYTEK Research(讯飞研究院)
Ensuring contextual faithfulness in retrieval-augmented large language models (LLMs) is crucial for building trustworthy information-seeking systems, particularly in long-form question-answering (LFQA) scenarios. In this work, we identify a salient correlation between LFQA faithfulness and retrieval heads, a set of attention heads responsible for retrieving contextual information. Leveraging this insight, we propose RHIO, a framework designed to teach LLMs to explicitly discriminate between faithful and unfaithful generations. RHIO first augments unfaithful samples that simulate realistic model-intrinsic errors by selectively masking retrieval heads. Then, these samples are incorporated into joint training, enabling the model to distinguish unfaithful outputs from faithful ones conditioned on control tokens. Furthermore, these control tokens are leveraged to self-induce contrastive outputs, amplifying their difference through contrastive decoding. Additionally, to facilitate the evaluation of contextual faithfulness, we also introduce GroundBench, a comprehensive benchmark compiled from five existing LFQA datasets. Extensive experimental results on GroundBench demonstrate that RHIO significantly improves faithfulness, even outperforming GPT-4o.
Investigating the Interplay between Contextual and Parametric Chain-of-Thought Faithfulness under Optimization
University of Copenhagen(哥本哈根大学);Technische Universität Berlin(柏林技术大学);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心(DFKI));BIFOLD – Berlin Institute for the Foundations of Learning and Data(BIFOLD – 柏林学习与数据基础研究院)
Exploiting Contextual Knowledge in LLMs through V-usable Information based Layer Enhancement
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences(认知与决策智能复杂系统重点实验室,自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Meituan(美团);University of Electronic Science and Technology of China(电子科技大学)
When Less Is Enough: Context Selection and Prompting Strategies for Bengali News Headline Generation
Charles Sturt University(查尔斯斯特大学);Chittagong University of Engineering and Technology(吉大港工程技术大学);Bangladesh Army University of Science and Technology(孟加拉国陆军理工大学)
School of Computer Science, University of Sydney(悉尼大学计算机科学学院);City University of Hong Kong(香港城市大学);Shanghai Jiao Tong University(上海交通大学);University of Oxford, Department of Engineering Science(牛津大学工程科学系);Department of Engineering Science, University of Oxford(牛津大学工程科学系)
Copy-Paste to Mitigate Large Language Model Hallucinations
Department of Computer Science, Tianjin University of Technology(天津理工大学计算机学院);National Institute of Health Data Science, Peking University(北京大学国家健康数据科学研究院);Tencent Jarvis Lab(腾讯Jarvis实验室)
Mixture of In-Context Experts Enhance LLMs' Long Context Awareness
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);XiaoMi AI Lab(小米AI实验室);Career Science Lab, BOSS Zhipin(BOSS直聘职业科学实验室);NLP Center, BOSS Zhipin(BOSS直聘自然语言处理中心)
LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Data
IDEA Research, International Digital Economy Academy(IDEA研究院,国际数字经济学院);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));DataArc Tech Ltd.(DataArc科技有限公司);Hithink RoyalFlush Information Network Co., Ltd(Hithink皇家Flush信息网络有限公司)
DeCoRe: Decoding by Contrasting Retrieval Heads to Mitigate Hallucinations
University of Edinburgh(爱丁堡大学);Centre for AI, Data Science & Artificial Intelligence, R&D, AstraZeneca(阿斯利康研发中心人工智能、数据科学与人工智能中心);University College London(伦敦大学学院)
METER: Evaluating Multi-Level Contextual Causal Reasoning in Large Language Models
College of Computer Science, Sichuan University(四川大学计算机学院);Institute of Data Science, National University of Singapore(新加坡国立大学数据科学研究所);Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, China(中国教育部机器学习与产业智能工程研究中心)
Hallucinations as Orthogonal Noise: Inference-Time Manifold Alignment via Dynamic Contextual Orthogonalization
Xi’an Jiaotong University(西安交通大学);Xingchen AGI Lab(星辰AGI实验室);China Telecom AI Technology (Beijing) Co., Ltd.(中国电信人工智能技术(北京)有限公司);Institute of Artificial Intelligence, China Telecom(中国电信人工智能研究院);University of Science and Technology of China(中国科学技术大学);Tsinghua University(清华大学)
The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentation
Department of Artificial Intelligence, Fraunhofer Heinrich Hertz Institute(人工智能系,弗劳恩霍夫海因里希·赫兹研究所);Department of Electrical Engineering and Computer Science, Technische Universität Berlin(电气工程与计算机科学系,柏林技术大学);BIFOLD - Berlin Institute for the Foundations of Learning and Data(柏林学习与数据基础研究所);Centre of eXplainable Artificial Intelligence, Technological University Dublin(可解释人工智能中心,都柏林技术大学)
Beyond Black-Box Interventions: Latent Probing for Faithful Retrieval-Augmented Generation
School of Informatics, Xiamen University(厦门大学信息学院);The Hong Kong Polytechnic University(香港理工大学);University of Chinese Academy of Sciences(中国科学院大学);Alibaba Group(阿里巴巴集团);Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China(福建省和台湾非物质文化遗产数字化保护与智能处理重点实验室(厦门大学),中华人民共和国文化和旅游部,中国)
BeliN: A Novel Corpus for Bengali Religious News Headline Generation using Contextual Feature Fusion
Chittagong University of Engineering and Technology(吉大港工程技术大学);Bangladesh Army University of Science and Technology (BAUST)(孟加拉国陆军科技大学);Charles Sturt University(查尔斯斯特大学)
An Attention Matrix for Every Decision: Faithfulness-based Arbitration Among Multiple Attention-Based Interpretations of Transformers in Text Classification
Aristotle University of Thessaloniki(塞萨洛尼基亚里士多德大学)