Recent studies have identified "retrieval heads" in Large Language Models (LLMs) responsible for extracting information from input contexts. However, prior works largely rely on static statistics aggregated across datasets, identifying heads that perform retrieval on average. This perspective overlooks the fine-grained temporal dynamics of autoregressive generation. In this paper, we investigate retrieval heads from a dynamic perspective. Through extensive analysis, we establish three core claims: (1) Dynamism: Retrieval heads vary dynamically across timesteps; (2) Irreplaceability: Dynamic retrieval heads are specific at each timestep and cannot be effectively replaced by static retrieval heads; and (3) Correlation: The model's hidden state encodes a predictive signal for future retrieval head patterns, indicating an internal planning mechanism. We validate these findings on the Needle-in-a-Haystack task and a multi-hop QA task, and quantify the differences on the utility of dynamic and static retrieval heads in a Dynamic Retrieval-Augmented Generation framework. Our study provides new insights into the internal mechanisms of LLMs.
Improving Contextual Faithfulness of Large Language Models via Retrieval Heads-Induced Optimization
Harbin Institute of Technology(哈尔滨工业大学);Peng Cheng Laboratory(鹏城实验室);Northeastern University(东北大学);The University of Hong Kong(香港大学);iFLYTEK Research(讯飞研究院)
DeCoRe: Decoding by Contrasting Retrieval Heads to Mitigate Hallucinations
University of Edinburgh(爱丁堡大学);Centre for AI, Data Science & Artificial Intelligence, R&D, AstraZeneca(阿斯利康研发中心人工智能、数据科学与人工智能中心);University College London(伦敦大学学院)
The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentation
Department of Artificial Intelligence, Fraunhofer Heinrich Hertz Institute(人工智能系,弗劳恩霍夫海因里希·赫兹研究所);Department of Electrical Engineering and Computer Science, Technische Universität Berlin(电气工程与计算机科学系,柏林技术大学);BIFOLD - Berlin Institute for the Foundations of Learning and Data(柏林学习与数据基础研究所);Centre of eXplainable Artificial Intelligence, Technological University Dublin(可解释人工智能中心,都柏林技术大学)
XL-HeadTags: Leveraging Multimodal Retrieval Augmentation for the Multilingual Generation of News Headlines and Tags
Ahsanullah University of Science & Technology(阿萨努拉科技大学);University of Alberta(阿尔伯塔大学);Université de Sherbrooke(谢布鲁克大学);Salesforce Research(Salesforce研究院);Nanyang Technological University(南洋理工大学)
HeteroCache: A Dynamic Retrieval Approach to Heterogeneous KV Cache Compression for Long-Context LLM Inference
School of Software Technology, Zhejiang University(浙江大学软件学院);China Mobile (Zhejiang) Research & Innovation Institute(中国移动(浙江)研究院);State Key Lab of CAD&CG, Zhejiang University(浙江大学CAD与CG国家重点实验室);Geely Automobile Research Institute (Ningbo) Co., Ltd.(吉利汽车研究院(宁波)有限公司);FABU Inc.(FABU公司)
SEAL: Scaling to Emphasize Attention for Long-Context Retrieval
Department of Convergence IT Engineering(融合IT工程系);Department of Computer Science and Engineering(计算机科学与工程系);Graduate School of Artificial Intelligence(人工智能研究生院)
Which Heads Matter for Reasoning? RL-Guided KV Cache Compression
Westlake University(西华大学);McGill University(麦吉尔大学);Mila - Quebec AI Institute(魁北克AI研究院);Zhejiang University(浙江大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德智能大学)
Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Beijing University of Posts and Telecommunications(北京邮电大学);Baidu Inc.(百度公司)
Query-focused and Memory-aware Reranker for Long Context Processing
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);Pattern Recognition Center, WeChat AI, Tencent(腾讯微信人工智能研究院模式识别中心);East China Normal University(华东师范大学)
Learning Hard Retrieval Decoder Attention for Transformers
DFKI(德国人工智能研究中心);Saarland University(萨尔大学);China Mobile Online Services(中国移动在线服务有限公司);Tianjin University(天津大学);Global Tone Communication Technology Co., Ltd.(全球语通通信技术有限公司)
Multi-Head RAG: Solving Multi-Aspect Problems with LLMs
Department of Computer Science, ETH Zurich(苏黎世联邦理工学院计算机科学系);IDEAS Research Institute(IDEAS研究 institute);NASK National Research Institute(国家研究 institute)
Think Parallax: Solving Multi-Hop Problems via Multi-View Knowledge-Graph-Based Retrieval-Augmented Generation
Yangtze Delta Region Institute (Huzhou), University of Electronic Science and Technology of China(长江三角洲地区研究院(湖州),电子科学与技术大学);School of Information and Software Engineering, University of Electronic Science and Technology of China(信息与软件工程学院,电子科学与技术大学)
ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretability
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Kuaishou Technology Co., Ltd.(快手科技有限公司);School of Information Technology and Management, University of International Business and Economics(对外经济贸易大学信息学院)