Large Language Models (LLMs) deployed on edge devices learn through fine-tuning and updating a certain portion of their parameters. Although such learning methods can be optimized to reduce resource utilization, the overall required resources remain a heavy burden on edge devices. Instead, Retrieval-Augmented Generation (RAG), a resource-efficient LLM learning method, can improve the quality of the LLM-generated content without updating model parameters. However, the RAG-based LLM may involve repetitive searches on the profile data in every user-LLM interaction. This search can lead to significant latency along with the accumulation of user data. Conventional efforts to decrease latency result in restricting the size of saved user data, thus reducing the scalability of RAG as user data continuously grows. It remains an open question: how to free RAG from the constraints of latency and scalability on edge devices? In this paper, we propose a novel framework to accelerate RAG via Computing-in-Memory (CiM) architectures. It accelerates matrix multiplications by performing in-situ computation inside the memory while avoiding the expensive data transfer between the computing unit and memory. Our framework, Robust CiM-backed RAG (RoCR), utilizing a novel contrastive learning-based training method and noise-aware training, can enable RAG to efficiently search profile data with CiM. To the best of our knowledge, this is the first work utilizing CiM to accelerate RAG.
A Systematic Study of Retrieval Pipeline Design for Retrieval-Augmented Medical Question Answering
Department of Mechatronics & Industrial Engineering, Chittagong University of Engineering & Technology(吉大港工程与技术大学机电与工业工程系);Department of Mechanical Engineering, Chittagong University of Engineering & Technology(吉大港工程与技术大学机械工程系)
Retrieval-Augmented Large Language Models for Evidence-Informed Guidance on Cannabidiol Use in Older Adults
Lawrence Bloomberg Faculty of Nursing, University of Toronto(多伦多大学劳伦斯·布隆伯格护理学院);KITE Research Institute, Toronto Rehabilitation Institute, University Health Network(大学健康网络多伦多康复研究所KITE研究所);College of Engineering and Technology, American University of the Middle East(中东美国大学工程与技术学院)
R^2AG: Incorporating Retrieval Information into Retrieval Augmented Generation
School of Computer Science, South China Normal University(华南师范大学计算机学院);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学)
Beyond the Parameters: A Technical Survey of Contextual Enrichment in Large Language Models: From In-Context Prompting to Causal Retrieval-Augmented Generation
Improving Context Fidelity via Native Retrieval-Augmented Reasoning
DIRO, Université de Montréal(蒙特利尔大学计算机科学与运筹学系);MetaGPT;Mila - Quebec AI Institute(米拉-魁北克人工智能研究所);McGill University(麦吉尔大学);Yale University(耶鲁大学);Canada CIFAR AI Chair(加拿大CIFAR人工智能教席)
Remote Sensing Retrieval-Augmented Generation: Bridging Remote Sensing Imagery and Comprehensive Knowledge with a Multi-Modal Dataset and Retrieval-Augmented Generation Model
School of Cyber Science and Technology, University of Science and Technology of China(信息科学技术学院,中国科学技术大学);China Academy of Electronics and Information Technology(电子信息技术研究院)
Long Context Modeling with Ranked Memory-Augmented Retrieval
University of New South Wales(新南威尔士大学);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学);University of Southampton(南安普顿大学);Mohamed Bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)