Retrieval-Augmented Generation (RAG) has emerged as a prominent method for incorporating domain knowledge into Large Language Models (LLMs). While RAG enhances response relevance by incorporating retrieved domain knowledge in the context, retrieval errors can still lead to hallucinations and incorrect answers. To recover from retriever failures, domain knowledge is injected by fine-tuning the model to generate the correct response, even in the case of retrieval errors. However, we observe that without systematic knowledge augmentation, fine-tuned LLMs may memorize new information but still fail to extract relevant domain knowledge, leading to poor performance. In this work, we present a novel framework that significantly enhances the fine-tuning process by augmenting the training data in two ways -- context augmentation and knowledge paraphrasing. In context augmentation, we create multiple training samples for a given QA pair by varying the relevance of the retrieved information, teaching the model when to ignore and when to rely on retrieved content. In knowledge paraphrasing, we fine-tune with multiple answers to the same question, enabling LLMs to better internalize specialized knowledge. To mitigate catastrophic forgetting due to fine-tuning, we add a domain-specific identifier to a question and also utilize a replay buffer containing general QA pairs. Experimental results demonstrate the efficacy of our method over existing techniques, achieving up to 10\% relative gain in token-level recall while preserving the LLM's generalization capabilities.
Comparing Knowledge Injection Methods for LLMs in a Low-Resource Regime
Faculdade de Engenharia Elétrica e de Computação (FEEC), University of Campinas (Unicamp)(电子工程与计算学院(FEEC),坎皮纳斯大学);Instituto de Computação (IC), University of Campinas (Unicamp)(计算研究所(IC),坎皮纳斯大学)
Retrieval-Augmented Generation for Large Language Models: A Survey
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University(同济大学上海智能自主系统研究院);College of Design and Innovation, Tongji University(同济大学设计与创新学院)
Position: LLMs Must Use Functor-Based and RAG-Driven Bias Mitigation for Fairness
Knight Foundation School of Computing and Information Sciences(骑士基金会计算与信息科学学院);Florida International University(佛罗里达国际大学);College of Engineering(工程学院);University of South Florida(佛罗里达州立大学)
DACL-RAG: Data Augmentation Strategy with Curriculum Learning for Retrieval-Augmented Generation
School of Information Science and Technology, University of Science and Technology of China(信息科学与技术学院,中国科学技术大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(人工智能研究院,合肥综合性国家科学中心)
Skill-RAG: Failure-State-Aware Retrieval Augmentation via Hidden-State Probing and Skill Routing
University of Michigan(密歇根大学);University of British Columbia(不列颠哥伦比亚大学);Rutgers University(罗格斯大学);University of Pennsylvania(宾夕法尼亚大学);New Jersey Institute of Technology(新泽西理工学院);TU Darmstadt(图腾斯大学);Wake Forest University(威克森林大学)
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Huawei Poisson Lab(华为泊松实验室);Waseda University(早稻田大学)
HiQA: A Hierarchical Contextual Augmentation RAG for Multi-Documents QA
Nanjing University of Aeronautics and Astronautics(南京航空航天大学);Southeast University(东南大学);Hello World(Shanghai) Technology Co., Ltd.(Hello World(上海)科技有限公司)
Injecting External Knowledge into the Reasoning Process Enhances Retrieval-Augmented Generation
State Key Laboratory of AI Safety(人工智能安全国家重点实验室);Institute of Computing Technology(计算技术研究所);Chinese Academy of Sciences(中国科学院);University of Chinese Academy of Sciences(中国科学院大学)
FCPRAG: Fusion-Controller Parametric Retrieval-Augmented Generation for Stable Multi-Passage LoRA Injection
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));University of Macau(澳门大学);Institute of Artificial Intelligence (TeleAI), China Telecom(中国电信人工智能研究院(TeleAI))
RING: Retrieval-Internalized Generation for Continual Large-Scale Knowledge Injection
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS(中国科学院计算技术研究所人工智能安全国家重点实验室);University of Chinese Academy of Sciences(中国科学院大学);Xiaohongshu Inc.(小红书科技有限公司)
QChunker: Learning Question-Aware Text Chunking for Domain RAG via Multi-Agent Debate
School of Information, Renmin University of China(中国人民大学信息学院);School of Smart Governance, Renmin University of China(中国人民大学智能治理学院);School of Information, BRAIN, Renmin University of China(中国人民大学信息学院、BRAIN学院);School of Economics and Management, Tsinghua University(清华大学经济管理学院)
When Large Multimodal Models Confront Evolving Knowledge: Challenges and Explorations
University of Science and Technology of China(中国科学技术大学);State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能全国重点实验室(北京通用人工智能研究院));Shandong University(山东大学);Nanjing University(南京大学);Wuhan University(武汉大学);The University of Sydney(悉尼大学);Northeast Forestry University(东北林业大学);Beijing Institute of Technology(北京理工大学)
MPIB: A Benchmark for Medical Prompt Injection Attacks and Clinical Safety in LLMs
Seoul National University College of Medicine(首尔国立大学医学院);Seoul National University(首尔国立大学);Seoul National University College of Medicine, Seoul National University Hospital(首尔国立大学医学院、首尔国立大学医院)
Lab-AI: Using Retrieval Augmentation to Enhance Language Models for Personalized Lab Test Interpretation in Clinical Medicine
Department of Statistics(统计学系);Florida State University(佛罗里达州立大学);School of Information(信息学院);Oklahoma State University(俄克拉荷马州立大学);Morsani College of Medicine(医学学院);University of South Florida(佛罗里达大学);College of Social Work(社会工作学院);Department of Computer Science(计算机科学系);Emory University(埃默里大学)