Multimodal Large Language Models (MLLMs) have achieved notable performance in computer vision tasks that require reasoning across visual and textual modalities, yet their capabilities are limited to their pre-trained data, requiring extensive fine-tuning for updates. Recent researches have explored the use of In-Context Learning (ICL) to overcome these challenges by providing a set of demonstrating examples as context to augment MLLMs performance in several tasks, showing that many-shot ICL leads to substantial improvements compared to few-shot ICL. However, the reliance on numerous demonstrating examples and the limited MLLMs context windows presents significant obstacles. This paper aims to address these challenges by introducing a novel approach, Visual RAG, that synergically combines the MLLMs capability to learn from the context, with a retrieval mechanism. The crux of this approach is to ensure to augment the MLLM knowledge by selecting only the most relevant demonstrating examples for the query, pushing it to learn by analogy. In this way, relying on the new information provided dynamically during inference time, the resulting system is not limited to the knowledge extracted from the training data, but can be updated rapidly and easily without fine-tuning. Furthermore, this greatly reduces the computational costs for improving the model image classification performance, and augments the model knowledge to new visual domains and tasks it was not trained for. Extensive experiments on eight different datasets in the state of the art spanning several domains and image classification tasks show that the proposed Visual RAG, compared to the most recent state of the art (i.e., many-shot ICL), is able to obtain an accuracy that is very close or even higher (approx. +2% improvement on average) while using a much smaller set of demonstrating examples (approx. only 23% on average).
Hybrid RAG-empowered Multi-modal LLM for Secure Data Management in Internet of Medical Things: A Diffusion-based Contract Approach
Guangdong University of Technology(广东工业大学);Nanjing University of Aeronautics and Astronautics(南京航空航天大学);Guangzhou University of Traditional Chinese Medicine(广州中医药大学);The Second Affiliated Hospital of Guangzhou University of Chinese Medicine(广州中医药大学第二附属医院)
Expanding the Boundaries of Vision Prior Knowledge in Multi-modal Large Language Models
University of Chinese Academy of Sciences(中国科学院大学);Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所信息处理实验室)
RAG-IGBench: Innovative Evaluation for RAG-based Interleaved Generation in Open-domain Question Answering
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China(认知智能国家重点实验室,中国科学技术大学);Xiaohongshu Inc.(小红书公司);Xi’an Jiaotong University(西安交通大学)
Benchmarking Multimodal RAG through a Chart-based Document Question-Answering Generation Framework
College of Computer Science, Chongqing University(重庆大学计算机学院);Aerospace Information Research Institute, Chinese Academy of Sciences(中国科学院空天信息创新研究院);Kuaishou Technology(快手科技)
Multimodal Information Fusion for Chart Understanding: A Survey of MLLMs -- Evolution, Limitations, and Cognitive Enhancement
College of Computer Science, Sichuan University(四川大学计算机科学学院);Engineering Research Center of Machine Learning(机器学习工程研究中心);China Telecom Institute of AI(中国电信人工智能研究院)
FedMLLM: Federated Fine-tuning MLLM on Multimodal Heterogeneity Data
Nanjing University of Science and Technology(南京理工大学);National University of Singapore(新加坡国立大学);Institute of High-Performance Computing, A*STAR(新加坡高性能计算研究院(新加坡科研局下属机构))
Large Language Model Aided Birt-Hogg-Dube Syndrome Diagnosis with Multimodal Retrieval-Augmented Generation
School of Computer Science and Technology(计算机科学与技术学院);Department of Pulmonary and Critical Care Medicine(呼吸与危重症医学科);Center for Diagnosis and Management of Rare Diseases(罕见病诊断与管理中心);the First Affiliated Hospital of USTC(USTC第一附属医院);Division of Life Sciences and Medicine(生命科学与医学系);USTC;WanNan Medical College(皖南医学院)
Taming a Retrieval Framework to Read Images in Humanlike Manner for Augmenting Generation of MLLMs
Emory University(埃默里大学);University of Electronic Science and Technology of China(电子科技大学);University of Illinois Chicago(伊利诺伊大学香槟分校);The Hong Kong Polytechnic University(香港理工大学)
Argus Inspection: Do Multimodal Large Language Models Possess the Eye of Panoptes?
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The University of Hong Kong(香港大学);Shanghai Jiao Tong University(上海交通大学);The Hong Kong University of Science and Technology(香港科学与技术大学);Fudan University(复旦大学);Tsinghua University(清华大学)
Why Do MLLMs Struggle with Spatial Understanding? A Systematic Analysis from Data to Architecture
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Tsinghua University(清华大学);Harbin Institute of Technology(哈尔滨工业大学);Wuhan AI Research(武汉人工智能研究院);Institute of Microelectronics of the Chinese Academy of Sciences(中国科学院微电子研究所)
SMA: Who Said That? Auditing Membership Leakage in Semi-Black-box RAG Controlling
Sun Yat-Sen University(孙中山大学);Nanyang Technological University(南洋理工大学);University of Chinese Academy of Science(中国科学院大学);Zhongguancun Academy(中关村学院);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所)
Faculty of Computing, Harbin Institute of Technology(计算机学院,哈尔滨工业大学);Department of Computer Science and Technology, Harbin Institute of Technology(计算机科学与技术系,哈尔滨工业大学);Harbin Institute of Technology Suzhou Research Institute(哈尔滨工业大学苏州研究院长);Peng Cheng Laboratory, Shenzhen, China(鹏城实验室,深圳,中国)
Laboratoire Interdisciplinaire Carnot de Bourgogne, Université de Bourgogne(勃艮第大学卡诺跨学科实验室);National University of Computer and Emerging Sciences (NUCES)(国家计算机与新兴科学大学)