arXiv 2503.188542025-03-25cs.CVcs.AI

MC-LLaVA：多概念个性化视觉语言模型

MC-LLaVA: Multi-Concept Personalized Vision-Language Model

Ruichuan An, Sihan Yang, Ming Lu, Renrui Zhang, Kai Zeng, Yulin Luo, Jiajun Cao, Hao Liang, Ying Chen, Qi She, Shanghang Zhang, Wentao Zhang

备注：: I sincerely apologize for any inconvenience caused. We actually uploaded this paper to arXiv in November 2024, as arXiv:2411.11706. During this update, we did not consider the replacement operation of arXiv, which led to duplicate submissions. We have made modifications at the original address arXiv:2411.11706

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

MC-LLaVA提出首个多概念个性化视觉语言模型，通过多概念指令微调和个性化视觉提示，实现多概念识别与定位，并贡献高质量数据集，提升VLM的用户特定助手能力。

摘要：

当前的视觉语言模型（VLM）在视觉问答等多样任务中展现出卓越的能力。为了提升用户体验，近期研究探索了VLM的个性化，以理解用户提供的概念。然而，这些研究主要聚焦于单概念个性化，忽视了多概念的存在及其相互作用，这限制了其在现实世界中的适用性。本文提出了首个多概念个性化范式MC-LLaVA。具体而言，MC-LLaVA采用多概念指令微调策略，在单次训练步骤中有效整合多个概念。为降低联合训练的相关成本，我们提出了一种个性化文本提示，利用视觉令牌信息初始化概念令牌。此外，我们在推理阶段引入个性化视觉提示，聚合位置置信度图以增强识别和定位能力。为推动多概念个性化研究，我们进一步贡献了一个高质量的指令微调数据集。我们精心收集了电影中包含多个人物和物体的图像，并手动生成面向多概念场景的问答样本，具有卓越的多样性。全面的定性和定量实验表明，MC-LLaVA能够实现令人印象深刻的多概念个性化响应，为VLM成为更好的用户特定助手铺平了道路。代码和数据集将在https://github.com/arctanxarc/MC-LLaVA公开提供。

英文摘要：

Current vision-language models (VLMs) show exceptional abilities across diverse tasks, such as visual question answering. To enhance user experience, recent studies investigate VLM personalization to understand user-provided concepts. However, they mainly focus on single-concept personalization, neglecting the existence and interplay of multiple concepts, which limits real-world applicability. This paper proposes the first multi-concept personalization paradigm, MC-LLaVA. Specifically, MC-LLaVA employs a multi-concept instruction tuning strategy, effectively integrating multiple concepts in a single training step. To reduce the costs related to joint training, we propose a personalized textual prompt that uses visual token information to initialize concept tokens. Additionally, we introduce a personalized visual prompt during inference, aggregating location confidence maps for enhanced recognition and grounding capabilities. To advance multi-concept personalization research, we further contribute a high-quality instruction tuning dataset. We carefully collect images with multiple characters and objects from movies and manually generate question-answer samples for multi-concept scenarios, featuring superior diversity. Comprehensive qualitative and quantitative experiments demonstrate that MC-LLaVA can achieve impressive multi-concept personalized responses, paving the way for VLMs to become better user-specific assistants. The code and dataset will be publicly available at https://github.com/arctanxarc/MC-LLaVA}.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。