This paper aims to overcome the "lost-in-the-middle" challenge of large language models (LLMs). While recent advancements have successfully enabled LLMs to perform stable language modeling with up to 4 million tokens, the persistent difficulty faced by most LLMs in identifying relevant information situated in the middle of the context has not been adequately tackled. To address this problem, this paper introduces Multi-scale Positional Encoding (Ms-PoE) which is a simple yet effective plug-and-play approach to enhance the capacity of LLMs to handle the relevant information located in the middle of the context, without fine-tuning or introducing any additional overhead. Ms-PoE leverages the position indice rescaling to relieve the long-term decay effect introduced by RoPE, while meticulously assigning distinct scaling ratios to different attention heads to preserve essential knowledge learned during the pre-training step, forming a multi-scale context fusion from short to long distance. Extensive experiments with a wide range of LLMs demonstrate the efficacy of our approach. Notably, Ms-PoE achieves an average accuracy gain of up to 3.8 on the Zero-SCROLLS benchmark over the original LLMs. Code are available at https://github.com/VITA-Group/Ms-PoE.
Listening to the Wise Few: Query-Key Alignment Unlocks Latent Correct Answers in Large Language Models
Skolkovo Institute of Science and Technology(斯科尔科沃科学技术研究院);AI Foundation and Algorithm Lab(人工智能基金会与算法实验室);Moscow Institute of Physics and Technology(莫斯科物理技术学院);Artificial Intelligence Research Institute (AIRI)(人工智能研究院);CNRS(法国国家科学研究中心);Université Paris Cité(巴黎西岱大学)
A Systematic Evaluation of Positional Bias in Multi-Video Summarization with MLLMs
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, Jilin University(知识驱动人机智能工程研究中心);International Center of Future Science, Jilin University(未来科学国际中心)
DSAS: A Universal Plug-and-Play Framework for Attention Optimization in Multi-Document Question Answering
Institute of Intelligent Computing, University of Electronic Science and Technology of China(电子科技大学智能计算研究院);School of Information and Software Engineering, University of Electronic Science and Technology of China(电子科技大学信息与软件工程学院)
LongHeads: Multi-Head Attention is Secretly a Long Context Processor
School of Computer Science, Fudan University(复旦大学计算机科学学院);Institute of Modern Languages and Linguistics, Fudan University(复旦大学现代语言与语言学研究所);International Human Phenome Institutes, Shanghai, China(国际人类表型组研究院(上海))
Dwell in the Beginning: How Language Models Embed Long Documents for Dense Retrieval
Language Technologies Institute, Carnegie Mellon University(卡内基梅隆大学语言技术研究所);Instituto Superior Técnico and INESC-ID, University of Lisbon(里斯本大学高级技术学院与INESC-ID研究所);NOVA LINCS, NOVA School of Science and Technology(NOVA科技大学NOVA LINCS研究所)
LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answering
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Zhipu AI(智谱AI);Tsinghua University(清华大学)