用上下文提示调优个性化你的大型视觉语言模型
Personalize Your Large Vision-language Models With In-context Prompt Tuning
AI总结：
提出上下文提示调优（ICPT）方法，通过轻量投影模块从多参考图像中提取细粒度视觉语义并转化为连续提示，结合几何正则化解决环境偏差和跨概念干扰，实现高效个性化。
摘要：
大型视觉语言模型（LVLMs）展示了强大的通用多模态能力，并越来越多地部署在下游系统中。这一趋势推动了对LVLM个性化的日益增长的兴趣，其目标是使模型能够快速有效地学习分布外的多模态概念，以满足用户特定需求。然而，许多现有方法依赖于推理时训练，降低了效率。它们也难以在复杂的多图像、多概念设置中保持准确性。这些限制制约了基于LVLM的系统的更广泛部署。因此，本文提出了上下文提示调优（ICPT）。具体来说，ICPT采用了一个轻量级投影模块，能够在复杂场景中操作，从多个参考图像中提取细粒度视觉语义，并将这些特征与身份标签映射无缝地转化为连续提示。为了最大化计算效率，该模块根据每个概念的内在视觉复杂性自适应地确定提示长度。关键的是，为了克服实际应用中普遍存在的环境偏差和跨概念干扰，我们引入了两种新颖的几何正则化。这些约束通过将关键身份与瞬态环境状态解耦，并分离概念以避免语义混淆，来优化提示表示。大量实验表明，ICPT在多种任务和LVLM骨干网络上实现了最先进的个性化准确性。
英文摘要：
Large vision-language models (LVLMs) have demonstrated strong general multimodal capability and are increasingly deployed in downstream systems. This trend has driven growing interest in LVLM personalization, which aims to enable models to quickly and effectively learn out-of-distribution multimodal concepts to meet user-specific needs. However, many existing methods rely on inference-time training, which reduces efficiency. They also struggle to maintain accuracy in complex multi-image, multi-concept settings. These limitations restrict the broader deployment of LVLM-based systems. Therefore, this paper proposes in-context prompt tuning (ICPT). Specifically, ICPT employs a lightweight projection module capable of operating in complex scenarios to extract fine-grained visual semantics from multiple reference images, seamlessly transforming these features alongside identity-label mappings into continuous prompts. To maximize computational efficiency, this module adaptively determines the prompt length based on the intrinsic visual complexity of each concept. Crucially, to overcome the environmental biases and cross-concept interference prevalent in real-world applications, we introduce two novel geometric regularizations. These constraints refine prompt representations by decoupling key identities from transient environmental states and separating concepts to avoid semantic confusion. Extensive experiments show that ICPT achieves state-of-the-art personalization accuracy across diverse tasks and LVLM backbones.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
