arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.11162cs.CL
收藏

检索头是动态的

Retrieval Heads are Dynamic

Michigan State University(密歇根州立大学);Zoom Communications(Zoom公司);Tongyi Lab, Alibaba Group(阿里云实验室)

Yuping Lin, Zitao Li, Yue Xing, Pengfei He, Yingqian Cui, Yaliang Li, Bolin Ding, Jingren Zhou, Jiliang Tang

AI总结:

本文从动态视角研究LLM中的检索头,揭示其时间动态性、不可替代性及与隐藏状态的关联,通过实验验证动态检索头在生成任务中的优势。

摘要:

近期研究表明,大型语言模型(LLM)中存在负责从输入上下文中提取信息的

英文摘要:

Recent studies have identified "retrieval heads" in Large Language Models (LLMs) responsible for extracting information from input contexts. However, prior works largely rely on static statistics aggregated across datasets, identifying heads that perform retrieval on average. This perspective overlooks the fine-grained temporal dynamics of autoregressive generation. In this paper, we investigate retrieval heads from a dynamic perspective. Through extensive analysis, we establish three core claims: (1) Dynamism: Retrieval heads vary dynamically across timesteps; (2) Irreplaceability: Dynamic retrieval heads are specific at each timestep and cannot be effectively replaced by static retrieval heads; and (3) Correlation: The model's hidden state encodes a predictive signal for future retrieval head patterns, indicating an internal planning mechanism. We validate these findings on the Needle-in-a-Haystack task and a multi-hop QA task, and quantify the differences on the utility of dynamic and static retrieval heads in a Dynamic Retrieval-Augmented Generation framework. Our study provides new insights into the internal mechanisms of LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑