Extending context windows (i.e., Long Context, LC) and using retrievers to selectively access relevant information (i.e., Retrieval-Augmented Generation, RAG) are the two main strategies to enable LLMs to incorporate extremely long external contexts. This paper revisits recent studies on this topic, highlighting their key insights and discrepancies. We then provide a more comprehensive evaluation by filtering out questions answerable without external context, identifying the most effective retrieval methods, and expanding the datasets. We show that LC generally outperforms RAG in question-answering benchmarks, especially for Wikipedia-based questions. Summarization-based retrieval performs comparably to LC, while chunk-based retrieval lags behind. However, RAG has advantages in dialogue-based and general question queries. These insights underscore the trade-offs between RAG and LC strategies, offering guidance for future optimization of LLMs with external knowledge sources. We also provide an in-depth discussion on this topic, highlighting the overlooked importance of context relevance in existing studies.
Confident RAG: Enhancing the Performance of LLMs for Mathematics Question Answering through Multi-Embedding and Confidence Scoring
Faculty of Education, The University of Hong Kong(香港大学教育学院);Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology(香港科学与技术大学土木与环境工程系)
Exploring the Role of Knowledge Graph-Based RAG in Japanese Medical Question Answering with Small-Scale LLMs
The University of Tokyo(东京大学);NEC Laboratories America(NEC美国实验室);Dokkyo Medical University(多摩医学大学);Graduate School of Frontier Sciences, The University of Tokyo(东京大学前沿科学研究生院)
Enhancing Large Language Models (LLMs) for Telecommunications using Knowledge Graphs and Retrieval-Augmented Generation
School of Computer Science(计算机科学学院);Department of Electrical and Computer Engineering(电气与计算机工程系);Standards and Mobility Innovation Lab(标准与移动创新实验室);Samsung Research America(三星美国研究)
SGIC: A Self-Guided Iterative Calibration Framework for RAG
NLP 2 CT Lab, Department of Computer and Information Science, University of Macau(自然语言处理2CT实验室,计算机与信息科学系,澳门大学);Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China(计算与智能学院,哈尔滨工业大学深圳校区,中国)
The Impact of Ideological Discourses in RAG: A Case Study with COVID-19 Treatments
Wichita State University(威斯康星州立大学);Pontifícia Universidade Católica de São Paulo(圣保罗天主教大学);Institut national de la recherche scientifique(国家科学研究中心);Université du Québec en Outaouais(魁北克大学Outaouais)
No Free Lunch: Retrieval-Augmented Generation Undermines Fairness in LLMs, Even for Vigilant Users
School of Data Science(数据科学学院);University of Virginia(弗吉尼亚大学);School of Management(管理学院);University of Science and Technology of China(中国科学技术大学);Department of Computer Science(计算机科学系);School of Computing(计算学院);University of Georgia(佐治亚大学)
U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystack
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University(同济大学上海智能自主系统研究院);School of Computer Science, Fudan University(复旦大学计算机科学技术学院);College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics(南京航空航天大学人工智能学院);College of Design and Innovation, Tongji University(同济大学设计与创新学院)
Multi-Head RAG: Solving Multi-Aspect Problems with LLMs
Department of Computer Science, ETH Zurich(苏黎世联邦理工学院计算机科学系);IDEAS Research Institute(IDEAS研究 institute);NASK National Research Institute(国家研究 institute)
The Other Side of the Coin: Exploring Fairness in Retrieval-Augmented Generation
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China(认知智能国家重点实验室,中国科学技术大学);Nanyang Technological University(南洋理工大学)
Laboratoire Interdisciplinaire Carnot de Bourgogne, Université de Bourgogne(勃艮第大学卡诺跨学科实验室);National University of Computer and Emerging Sciences (NUCES)(国家计算机与新兴科学大学)