Language model pre-training has been shown to capture a surprising amount of world knowledge, crucial for NLP tasks such as question answering. However, this knowledge is stored implicitly in the parameters of a neural network, requiring ever-larger networks to cover more facts.
To capture knowledge in a more modular and interpretable way, we augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner, using masked language modeling as the learning signal and backpropagating through a retrieval step that considers millions of documents.
We demonstrate the effectiveness of Retrieval-Augmented Language Model pre-training (REALM) by fine-tuning on the challenging task of Open-domain Question Answering (Open-QA). We compare against state-of-the-art models for both explicit and implicit knowledge storage on three popular Open-QA benchmarks, and find that we outperform all previous methods by a significant margin (4-16% absolute accuracy), while also providing qualitative benefits such as interpretability and modularity.
A Systematic Study of Retrieval Pipeline Design for Retrieval-Augmented Medical Question Answering
Department of Mechatronics & Industrial Engineering, Chittagong University of Engineering & Technology（吉大港工程与技术大学机电与工业工程系）；Department of Mechanical Engineering, Chittagong University of Engineering & Technology（吉大港工程与技术大学机械工程系）
Enhancing Retrieval-Augmented LMs with a Two-stage Consistency Learning Compressor
College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）；Artificial Intelligence Laboratory, China Mobile Communication Group Tianjin Co., Ltd.（中国移动通信集团天津有限公司人工智能实验室）；University College London（伦敦大学学院）
DioR: Adaptive Cognitive Detection and Contextual Retrieval Optimization for Dynamic Retrieval-Augmented Generation
Zhejiang Key Laboratory of Intelligent Education Technology and Application, Zhejiang Normal University（浙江智能教育技术与应用重点实验室，浙江师范大学）；School of Computer Science and Technology, Zhejiang Normal University（浙江师范大学计算机科学与技术学院）；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Department of Computer Science and Engineering, Hong Kong University of Science and Technology（香港科技大学计算机科学与工程系）；School of Computer Science and Technology, Soochow University（苏州大学计算机科学与技术学院）
RAR$^2$: Retrieval-Augmented Medical Reasoning via Thought-Driven Retrieval
Department of Computing, The Hong Kong Polytechnic University（计算机系，香港理工大学）；Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering, Southern University of Science and Technology（可信自主系统研究院和计算机科学与工程系，南方科技大学）
ParetoRAG: Leveraging Sentence-Context Attention for Robust and Efficient Retrieval-Augmented Generation
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cybersecurity, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Alibaba Group（阿里巴巴集团）
RPO: Retrieval Preference Optimization for Robust Retrieval-Augmented Generation
National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China（语音与语言信息处理国家工程研究中心，中国科学技术大学）；State Key Laboratory of Cognitive Intelligence, iFLYTEK Research（认知智能国家重点实验室，iFLYTEK研究院）
Outside-Knowledge Visual Question Answering (OK-VQA) is a challenging VQA task that requires retrieval of external knowledge to answer questions about images. Recent OK-VQA systems use Dense Passage Retrieval (DPR) to retrieve documents from external knowledge bases, such as Wikipedia, but with DPR trained separately from answer generation, introducing a potential limit on the overall system performance. Instead, we propose a joint training scheme which includes differentiable DPR integrated with answer generation so that the system can be trained in an end-to-end fashion. Our experiments show…
GraphSearch: An Agentic Deep Searching Workflow for Graph Retrieval-Augmented Generation
IDEA Research, International Digital Economy Academy（IDEA研究机构、国际数字经济学院）；Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；DataArc Tech Ltd.（DataArc科技有限公司）；Hithink RoyalFlush Information Network Co., Ltd（Hithink皇家Flush信息网络有限公司）
LevelRAG: Enhancing Retrieval-Augmented Generation with Multi-hop Logic Planning over Rewriting Augmented Searchers
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Chinese Academy of Sciences（中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
A Large Language Model-based Framework for Semi-Structured Tender Document Retrieval-Augmented Generation
School of Information Management Sun Yat-sen University Guangzhou, China；School of Information Management Sun Yat-sen University Guangzhou, China corresponding author
University of Illinois, Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Microsoft Azure AI（微软Azure人工智能部门）；Microsoft Research（微软研究院）
We propose Generation-Augmented Retrieval (GAR) for answering open-domain questions, which augments a query through text generation of heuristically discovered relevant contexts without external resources as supervision. We demonstrate that the generated contexts substantially enrich the semantics of the queries and GAR with sparse representations (BM25) achieves comparable or better performance than state-of-the-art dense retrieval methods such as DPR. We show that generating diverse contexts for a query is beneficial as fusing their results consistently yields better retrieval accuracy. More…
