Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization
University of Washington(华盛顿大学);MIT(麻省理工学院);Google Cloud AI Research(谷歌云人工智能研究院);Google DeepMind(谷歌DeepMind)
Cheng-Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang, Long T. Le, Abhishek Kumar, James Glass, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, Tomas Pfister
Large language models (LLMs), even when specifically trained to process long input contexts, struggle to capture relevant information located in the middle of their input. This phenomenon has been known as the lost-in-the-middle problem. In this work, we make three contributions. First, we set out to understand the factors that cause this phenomenon. In doing so, we establish a connection between lost-in-the-middle to LLMs' intrinsic attention bias: LLMs exhibit a U-shaped attention bias where the tokens at the beginning and at the end of its input receive higher attention, regardless of their relevance. Second, we mitigate this positional bias through a calibration mechanism, found-in-the-middle, that allows the model to attend to contexts faithfully according to their relevance, even though when they are in the middle. Third, we show found-in-the-middle not only achieves better performance in locating relevant information within a long context, but also eventually leads to improved retrieval-augmented generation (RAG) performance across various tasks, outperforming existing methods by up to 15 percentage points. These findings open up future directions in understanding LLM attention bias and its potential consequences.
Listening to the Wise Few: Query-Key Alignment Unlocks Latent Correct Answers in Large Language Models
Skolkovo Institute of Science and Technology(斯科尔科沃科学技术研究院);AI Foundation and Algorithm Lab(人工智能基金会与算法实验室);Moscow Institute of Physics and Technology(莫斯科物理技术学院);Artificial Intelligence Research Institute (AIRI)(人工智能研究院);CNRS(法国国家科学研究中心);Université Paris Cité(巴黎西岱大学)
A Systematic Evaluation of Positional Bias in Multi-Video Summarization with MLLMs
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, Jilin University(知识驱动人机智能工程研究中心);International Center of Future Science, Jilin University(未来科学国际中心)
Self-Describing Structured Data with Dual-Layer Guidance: A Lightweight Alternative to RAG for Precision Retrieval in Large-Scale LLM Knowledge Navigation
LongHeads: Multi-Head Attention is Secretly a Long Context Processor
School of Computer Science, Fudan University(复旦大学计算机科学学院);Institute of Modern Languages and Linguistics, Fudan University(复旦大学现代语言与语言学研究所);International Human Phenome Institutes, Shanghai, China(国际人类表型组研究院(上海))
Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrieval
Language Technologies Institute, Carnegie Mellon University(卡内基梅隆大学语言技术研究所);Instituto Superior Técnico and INESC-ID, University of Lisbon(里斯本大学高级技术学院与INESC-ID研究所);NOVA LINCS, NOVA School of Science and Technology(NOVA科技大学NOVA LINCS研究所)