Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, Luke Zettlemoyer, Scott Yih
Retrieval-augmented language models (RALMs) improve performance by accessing long-tail and up-to-date knowledge from external data stores, but are challenging to build. Existing approaches require either expensive retrieval-specific modifications to LM pre-training or use post-hoc integration of the data store that leads to suboptimal performance. We introduce Retrieval-Augmented Dual Instruction Tuning (RA-DIT), a lightweight fine-tuning methodology that provides a third option by retrofitting any LLM with retrieval capabilities. Our approach operates in two distinct fine-tuning steps: (1) one updates a pre-trained LM to better use retrieved information, while (2) the other updates the retriever to return more relevant results, as preferred by the LM. By fine-tuning over tasks that require both knowledge utilization and contextual awareness, we demonstrate that each stage yields significant performance improvements, and using both leads to additional gains. Our best model, RA-DIT 65B, achieves state-of-the-art performance across a range of knowledge-intensive zero- and few-shot learning benchmarks, significantly outperforming existing in-context RALM approaches by up to +8.9% in 0-shot setting and +1.4% in 5-shot setting on average.
NavRAG: Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLM
School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；School of Computing, National University of Singapore（新加坡国立大学计算机学院）；College of Information Science and Technology, Beijing University of Chemical Technology（北京化工大学信息科学与技术学院）
MIT Computer Science and Artificial Intelligence Lab（麻省理工学院计算机科学与人工智能实验室）；CUHK Centre for Perceptual and Interactive Intelligence（香港中文大学感知与交互智能中心）；MIT Linguistics（麻省理工学院语言学系）
Toward General Instruction-Following Alignment for Retrieval-Augmented Generation
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）
Instruction-Augmented Long-Horizon Planning: Embedding Grounding Mechanisms in Embodied Mobile Manipulation
The Hong Kong Polytechnic University（香港理工大学）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（东方理工大学宁波数字孪生研究院）；Great Bay University（大湾区大学）；Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Accelerating Adaptive Retrieval Augmented Generation via Instruction-Driven Representation Reduction of Retrieval Overlaps
School of Information and Software Engineering, University of Electronic Science and Technology of China（电子科技大学信息与软件工程学院）；Central South University（中南大学）；Big data and artificial intelligent institute, China Telecom Research Institute（中国电信大数据与人工智能研究院）
ZhiFangDanTai: Fine-tuning Graph-based Retrieval-Augmented Generation Model for Traditional Chinese Medicine Formula
School of Management, Capital Normal University（管理学院，首都师范大学）；School of Electrical Engineering and Computer Science, the University of Queensland（电气工程与计算机科学学院，昆士兰大学）
Domain-Specific Improvement on Psychotherapy Chatbot Using Assistant
Czech Technical University in Prague（布拉格捷克技术大学）；National Institute of Mental Health in Prague（布拉格国家精神卫生研究所）；The University of Hong Kong（香港大学）；Shenzhen Mental Health Centre（深圳市精神卫生中心）
oRetrieval Augmented Generation for 10 Large Language Models and its Generalizability in Assessing Medical Fitness
Singapore General Hospital（新加坡中央医院）；Duke-NUS Medical School（杜克-新加坡国立大学医学院）；Singapore National Eye Centre（新加坡国家眼科中心）；Singapore Eye Research Institute（新加坡眼科研究所）；Singapore Health Services（新加坡保健服务集团）；KK Women’s and Children’s Hospital（竹脚妇幼医院）；Chang Gung Memorial Hospital（长庚纪念医院）；Chang Gung University（长庚大学）；Kaohsiung Chang Gung Memorial Hospital（高雄长庚纪念医院）；National Sun Yat-sen University（国立中山大学）；Brigham and Women’s Hospital（布莱根和妇女医院）；Harvard Medical School（哈佛医学院）
Retrieval-Augmented and Knowledge-Grounded Language Models for Faithful Clinical Medicine
University of Oxford（牛津大学）；Peking University（北京大学）；Yale University（耶鲁大学）；Tencent（腾讯）；Oxford-Suzhou Centre for Advanced Research（牛津-苏州高等研究院）；Shanghai Jiao Tong University School of Medicine（上海交通大学医学院）
ProSwitch: Knowledge-Guided Instruction Tuning to Switch Between Professional and Non-Professional Responses
Zhejiang University of Science and Technology（浙江科技大学）；Fudan University（复旦大学）；Zhejiang University（浙江大学）；The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）
KAER: A Knowledge Augmented Pre-Trained Language Model for Entity Resolution
School of Information Sciences, University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校信息科学学院）；Informatics, University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校信息学系）
Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalization
Tsinghua University（清华大学）；Institute for Artificial Intelligence（人工智能研究院）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）
Robin3D: Improving 3D Large Language Model via Robust Instruction Tuning
University of Illinois Chicago（伊利诺伊大学芝加哥分校）；Zhejiang University（浙江大学）；University of Central Florida（中佛罗里达大学）；Illinois Institute of Technology（伊利诺伊理工大学）
From Classification to Generation: Insights into Crosslingual Retrieval Augmented ICL
Center for Information and Language Processing (CIS), LMU Munich（慕尼黑大学信息与语言处理中心（CIS））；Academy of Cyber（网络研究院）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心（MCML））
