Previous work finds that recent long-context language models fail to make equal use of information in the middle of their inputs, preferring pieces of information located at the tail ends which creates an undue bias in situations where we would like models to be equally capable of using different parts of the input. Thus far, the problem has mainly only been considered in settings with single pieces of critical information, leading us to question what happens when multiple necessary pieces of information are spread out over the inputs. Here, we demonstrate the effects of the "lost in the middle" problem in the multi-hop question answering setting -- in which multiple reasoning "hops" over disconnected documents are required -- and show that performance degrades not only with respect to the distance of information from the edges of the context, but also between pieces of information. Additionally, we experiment with means of alleviating the problem by reducing superfluous document contents through knowledge graph triple extraction and summarization, and prompting models to reason more thoroughly using chain-of-thought prompting.
Unleashing Multi-Hop Reasoning Potential in Large Language Models through Repetition of Misordered Context
Department of Electrical and Computer Engineering, Seoul National University(电子与计算机工程系,首尔国立大学);AIIS, ASRI, INMC, ISRC, and IPAI, Seoul National University(人工智能研究所、人工智能研究机构、智能网络中心、智能系统研究所以及人工智能与公共政策研究所,首尔国立大学)
Mitigating Lost-in-Retrieval Problems in Retrieval Augmented Multi-Hop Question Answering
State Key Laboratory for Novel Software Technology, Nanjing University, China(新型软件技术国家重点实验室,南京大学,中国);University of California, Merced, USA(加州大学默塞德分校,美国);National Institute of Healthcare Data Science, Nanjing University, China(健康数据科学国家研究院,南京大学,中国)
DSAS: A Universal Plug-and-Play Framework for Attention Optimization in Multi-Document Question Answering
Institute of Intelligent Computing, University of Electronic Science and Technology of China(电子科技大学智能计算研究院);School of Information and Software Engineering, University of Electronic Science and Technology of China(电子科技大学信息与软件工程学院)
A MapReduce Approach to Effectively Utilize Long Context Information in Retrieval Augmented Language Models
Columbia University(哥伦比亚大学);Weill Cornell Medicine(威尔康奈尔医学院);National Institutes of Health(美国国立卫生研究院);Weill Cornell Medical College(威尔康奈尔医学院);University of Texas Southwestern Medical Center(德克萨斯大学西南医学中心);NYU Grossman School of Medicine(纽约大学格罗斯曼医学院)
Optimizing Multi-Hop Document Retrieval Through Intermediate Representations
School of Software, BNRist, Tsinghua University(软件学院、BNRist、清华大学);Beijing Key Laboratory of Industrial Bigdata System and Application(北京工业大数据系统与应用重点实验室);Gaoling School of Artificial Intelligence, Renmin University of China(甘露人工智能学院、中国人民大学)
Self-Describing Structured Data with Dual-Layer Guidance: A Lightweight Alternative to RAG for Precision Retrieval in Large-Scale LLM Knowledge Navigation