Extending the context window of large language models (LLMs) is getting popular recently, while the solution of augmenting LLMs with retrieval has existed for years. The natural questions are: i) Retrieval-augmentation versus long context window, which one is better for downstream tasks? ii) Can both methods be combined to get the best of both worlds? In this work, we answer these questions by studying both solutions using two state-of-the-art pretrained LLMs, i.e., a proprietary 43B GPT and Llama2-70B. Perhaps surprisingly, we find that LLM with 4K context window using simple retrieval-augmentation at generation can achieve comparable performance to finetuned LLM with 16K context window via positional interpolation on long context tasks, while taking much less computation. More importantly, we demonstrate that retrieval can significantly improve the performance of LLMs regardless of their extended context window sizes. Our best model, retrieval-augmented Llama2-70B with 32K context window, outperforms GPT-3.5-turbo-16k and Davinci003 in terms of average score on nine long context tasks including question answering, query-based summarization, and in-context few-shot learning tasks. It also outperforms its non-retrieval Llama2-70B-32k baseline by a margin, while being much faster at generation. Our study provides general insights on the choice of retrieval-augmentation versus long context extension of LLM for practitioners.
SEAL: Scaling to Emphasize Attention for Long-Context Retrieval
Department of Convergence IT Engineering（融合IT工程系）；Department of Computer Science and Engineering（计算机科学与工程系）；Graduate School of Artificial Intelligence（人工智能研究生院）
Information Retrieval Meets Large Language Models: A Strategic Report from Chinese IR Community
Tsinghua University（清华大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Huawei Technologies Ltd. Co（华为技术有限公司）；Jilin University（吉林大学）；Zhejiang University（浙江大学）；Shandong University（山东大学）；Shandong Artificial Intelligence Institute（山东人工智能研究院）；South China University of Technology（华南理工大学）；Renmin University of China（中国人民大学）；University of Science and Technology of China（中国科学技术大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Wuhan University（武汉大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shanxi University（山西大学）；Jiangxi Normal University（江西师范大学）；Hefei University of Technology（合肥工业大学）；Baidu Inc.（百度公司）；Tianjin University（天津大学）；Shanghai Jiao Tong University（上海交通大学）；Chongqing University of Technology（重庆理工大学）
Investigating Pedagogical Teacher and Student LLM Agents: Genetic Adaptation Meets Retrieval Augmented Generation Across Learning Style
RespAI Lab, KIIT Bhubaneswar（KIIT布巴内斯瓦尔 RespAI 实验室）；BITS Pilani（皮拉尼比尔拉理工学院）；University of South Florida（南佛罗里达大学）；Pennsylvania State University（宾夕法尼亚州立大学）
The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentation
Department of Artificial Intelligence, Fraunhofer Heinrich Hertz Institute（人工智能系，弗劳恩霍夫海因里希·赫兹研究所）；Department of Electrical Engineering and Computer Science, Technische Universität Berlin（电气工程与计算机科学系，柏林技术大学）；BIFOLD - Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所）；Centre of eXplainable Artificial Intelligence, Technological University Dublin（可解释人工智能中心，都柏林技术大学）
Beyond the Parameters: A Technical Survey of Contextual Enrichment in Large Language Models: From In-Context Prompting to Causal Retrieval-Augmented Generation
Improving Context Fidelity via Native Retrieval-Augmented Reasoning
DIRO, Université de Montréal（蒙特利尔大学计算机科学与运筹学系）；MetaGPT；Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；McGill University（麦吉尔大学）；Yale University（耶鲁大学）；Canada CIFAR AI Chair（加拿大CIFAR人工智能教席）
Biomedical Hypothesis Explainability with Graph-Based Context Retrieval
Center for Bioinformatics and Computational Biology University of Delaware（生物信息学与计算生物学中心 大田纳西大学）；Computer and Information Sciences University of Delaware（计算机与信息科学 大田纳西大学）；Drug Discovery and Biomedical Sciences (DDBS) College of Pharmacy University of South Carolina（药物发现与生物医学科学（DDBS）药学院 美国南卡罗来纳大学）
