MC-LLaVA：多概念个性化视觉-语言模型
MC-LLaVA: Multi-Concept Personalized Vision-Language Model
AI总结：
MC-LLaVA通过多概念指令微调和个性化提示提升视觉-语言模型的多概念个性化能力。
摘要：
当前的视觉-语言模型（VLMs）在多种任务上表现出色，例如视觉问答。为了提升用户体验，最近的研究探讨了VLM的个性化以理解用户提供的概念。然而，它们主要集中在单一概念上，忽视了多个概念的存在及其相互作用，这限制了实际应用。本文提出了MC-LLaVA，一种多概念个性化范式。具体而言，MC-LLaVA采用多概念指令微调策略，有效在单个训练步骤中整合多个概念。为减少训练成本，我们提出了一种个性化文本提示，利用视觉令牌信息来初始化概念令牌。此外，我们在推理过程中引入了个性化视觉提示，聚合位置图以增强识别和定位能力。为进一步推动性能上限，我们加入了可选的辅助损失，更好地增强所提出的个性化提示。为了装饰VLM个性化研究，我们贡献了一个高质量的数据集。我们仔细收集了包含多个角色和物体的电影图像，并手动创建了多概念场景的问题-答案样本，具有优越的多样性。全面的实验表明，MC-LLaVA实现了令人印象深刻的多概念个性化响应，为VLM成为更好的用户助手铺平了道路。代码和数据集将在https://github.com/arctanxarc/MC-LLaVA上发布。
英文摘要：
Current vision-language models (VLMs) show exceptional abilities across diverse tasks, such as visual question answering. To enhance user experience, recent studies have investigated VLM personalization to understand user-provided concepts. However, they mainly focus on single concepts, neglecting the existence and interplay of multiple concepts, which limits real-world applicability. This paper proposes MC-LLaVA, a multi-concept personalization paradigm. Specifically, MC-LLaVA employs a multi-concept instruction tuning strategy, effectively integrating multiple concepts in a single training step. To reduce the training costs, we propose a personalized textual prompt that uses visual token information to initialize concept tokens. Additionally, we introduce a personalized visual prompt during inference, aggregating location maps for enhanced recognition and grounding capabilities. To further push the performance upper bound, we incorporate an optional auxiliary loss, better enhancing the proposed personalized prompts. To decorate the VLM personalization research, we contribute a high-quality dataset. We carefully collect images with multiple characters and objects from movies and manually create question-answer samples for multi-concept scenarios, featuring superior diversity. Comprehensive experiments demonstrate that MC-LLaVA achieves impressive multi-concept personalized responses, paving the way for VLMs to become better user assistants. The code and dataset will be released at \href{https://github.com/arctanxarc/MC-LLaVA}{https://github.com/arctanxarc/MC-LLaVA}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
