arXiv 2410.133602025-03-28cs.CVcs.AIcs.CLcs.LGcs.MM

RAP：面向多模态大语言模型的检索增强个性化

RAP: Retrieval-Augmented Personalization for Multimodal Large Language Models

The Chinese University of Hong Kong（香港中文大学）；Nanjing University（南京大学）；Beijing Institute of Technology（北京理工大学）；CUHK（香港中文大学）

Haoran Hao, Jiaming Han, Changsheng Li, Yu-Feng Li, Xiangyu Yue

备注：: Accepted by CVPR 2025. Code: https://github.com/Hoar012/RAP-MLLM

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出RAP框架，通过记忆、检索、生成三步将通用多模态大模型转变为个性化助手，支持实时概念编辑，无需微调即可泛化至无限视觉概念，在个性化图像描述、问答和视觉识别等任务中表现卓越。

摘要：

大语言模型（LLMs）的发展显著增强了多模态大语言模型（MLLMs）作为通用助手的性能。然而，缺乏用户特定知识仍然限制了它们在人类日常生活中的应用。在本文中，我们引入了检索增强个性化（RAP）框架，用于MLLMs的个性化。从通用MLLM开始，我们通过三个步骤将其转变为个性化助手。（a）记忆：我们设计了一个键值数据库来存储用户相关信息，例如用户的姓名、头像和其他属性。（b）检索：当用户发起对话时，RAP将使用多模态检索器从数据库中检索相关信息。（c）生成：输入查询和检索到的概念信息被馈送到MLLMs中，以生成个性化的、知识增强的响应。与先前方法不同，RAP通过更新外部数据库允许实时概念编辑。为了进一步提高生成质量和与用户特定信息的一致性，我们设计了一个数据收集流程，并创建了一个专门的数据集，用于MLLMs的个性化训练。基于该数据集，我们训练了一系列MLLMs作为个性化多模态助手。通过在大规模数据集上进行预训练，RAP-MLLMs能够泛化到无限的视觉概念，而无需额外的微调。我们的模型在各种任务中表现出卓越的灵活性和生成质量，例如个性化图像描述、问答和视觉识别。代码、数据和模型可在https://hoar012.github.io/RAP-Project/获取。

英文摘要：

The development of large language models (LLMs) has significantly enhanced the capabilities of multimodal LLMs (MLLMs) as general assistants. However, lack of user-specific knowledge still restricts their application in human's daily life. In this paper, we introduce the Retrieval Augmented Personalization (RAP) framework for MLLMs' personalization. Starting from a general MLLM, we turn it into a personalized assistant in three steps. (a) Remember: We design a key-value database to store user-related information, e.g., user's name, avatar and other attributes. (b) Retrieve: When the user initiates a conversation, RAP will retrieve relevant information from the database using a multimodal retriever. (c) Generate: The input query and retrieved concepts' information are fed into MLLMs to generate personalized, knowledge-augmented responses. Unlike previous methods, RAP allows real-time concept editing via updating the external database. To further improve generation quality and alignment with user-specific information, we design a pipeline for data collection and create a specialized dataset for personalized training of MLLMs. Based on the dataset, we train a series of MLLMs as personalized multimodal assistants. By pretraining on large-scale dataset, RAP-MLLMs can generalize to infinite visual concepts without additional finetuning. Our models demonstrate outstanding flexibility and generation quality across a variety of tasks, such as personalized image captioning, question answering and visual recognition. The code, data and models are available at https://hoar012.github.io/RAP-Project/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。