Retrieval-augmented generation (RAG) techniques have emerged as a promising solution to enhance the reliability of large language models (LLMs) by addressing issues like hallucinations, outdated knowledge, and domain adaptation. In particular, existing RAG methods append relevant documents retrieved from external corpus or databases to the input of LLMs to guide their generation process, which we refer to as the in-context knowledge injection method. While this approach is simple and often effective, it has inherent limitations. Firstly, increasing the context length and number of relevant documents can lead to higher computational overhead and degraded performance, especially in complex reasoning tasks. More importantly, in-context knowledge injection operates primarily at the input level, but LLMs store their internal knowledge in their parameters. This gap fundamentally limits the capacity of in-context methods. To this end, we introduce Parametric retrieval-augmented generation (Parametric RAG), a new RAG paradigm that integrates external knowledge directly into the parameters of feed-forward networks (FFN) of an LLM through document parameterization. This approach not only saves online computational costs by eliminating the need to inject multiple documents into the LLMs' input context, but also deepens the integration of external knowledge into the parametric knowledge space of the LLM. Experimental results demonstrate that Parametric RAG substantially enhances both the effectiveness and efficiency of knowledge augmentation in LLMs. Also, it can be combined with in-context RAG methods to achieve even better performance.
We have open-sourced all the code, data, and models in the following anonymized GitHub link: https://github.com/oneal2000/PRAG
LevelRAG: Enhancing Retrieval-Augmented Generation with Multi-hop Logic Planning over Rewriting Augmented Searchers
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Chinese Academy of Sciences(中国科学院);University of Chinese Academy of Sciences(中国科学院大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
A Large Language Model-based Framework for Semi-Structured Tender Document Retrieval-Augmented Generation
School of Information Management Sun Yat-sen University Guangzhou, China;School of Information Management Sun Yat-sen University Guangzhou, China corresponding author
CausalRAG: Integrating Causal Graphs into Retrieval-Augmented Generation
Department of Computer and Data Sciences, Case Western Reserve University(计算机与数据科学系,凯斯西储大学);Department of Design and Innovation, Case Western Reserve University(设计与创新系,凯斯西储大学)
R^2AG: Incorporating Retrieval Information into Retrieval Augmented Generation
School of Computer Science, South China Normal University(华南师范大学计算机学院);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学)
Semi-Parametric Retrieval via Binary Bag-of-Tokens Index
The Hong Kong University of Science and Technology(香港科技大学);Microsoft Research(微软研究院);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Enhancing Retrieval-Augmented LMs with a Two-stage Consistency Learning Compressor
College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部);Artificial Intelligence Laboratory, China Mobile Communication Group Tianjin Co., Ltd.(中国移动通信集团天津有限公司人工智能实验室);University College London(伦敦大学学院)
DioR: Adaptive Cognitive Detection and Contextual Retrieval Optimization for Dynamic Retrieval-Augmented Generation
Zhejiang Key Laboratory of Intelligent Education Technology and Application, Zhejiang Normal University(浙江智能教育技术与应用重点实验室,浙江师范大学);School of Computer Science and Technology, Zhejiang Normal University(浙江师范大学计算机科学与技术学院);School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院);Department of Computer Science and Engineering, Hong Kong University of Science and Technology(香港科技大学计算机科学与工程系);School of Computer Science and Technology, Soochow University(苏州大学计算机科学与技术学院)
RAR$^2$: Retrieval-Augmented Medical Reasoning via Thought-Driven Retrieval
Department of Computing, The Hong Kong Polytechnic University(计算机系,香港理工大学);Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering, Southern University of Science and Technology(可信自主系统研究院和计算机科学与工程系,南方科技大学)
ParetoRAG: Leveraging Sentence-Context Attention for Robust and Efficient Retrieval-Augmented Generation
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cybersecurity, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Alibaba Group(阿里巴巴集团)
RPO: Retrieval Preference Optimization for Robust Retrieval-Augmented Generation
National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China(语音与语言信息处理国家工程研究中心,中国科学技术大学);State Key Laboratory of Cognitive Intelligence, iFLYTEK Research(认知智能国家重点实验室,iFLYTEK研究院)
GraphSearch: An Agentic Deep Searching Workflow for Graph Retrieval-Augmented Generation
IDEA Research, International Digital Economy Academy(IDEA研究机构、国际数字经济学院);Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));DataArc Tech Ltd.(DataArc科技有限公司);Hithink RoyalFlush Information Network Co., Ltd(Hithink皇家Flush信息网络有限公司)