探索多模态大语言模型中视觉提示的可迁移性
Exploring the Transferability of Visual Prompting for Multimodal Large Language Models
AI总结：
针对多模态大语言模型下游任务适配成本高的问题，提出可迁移视觉提示方法，通过两种策略提升提示跨模型迁移能力，仅在单模型训练即可提升多种MLLMs的多类下游任务性能。
摘要：
尽管多模态大语言模型（Multimodal Large Language Models, MLLMs）已展现出颇具潜力的通用能力，但在下游任务上的性能仍不及专用模型，因此有必要进行适配以提升其实用性。然而，微调方法需要为每个模型单独训练，会带来巨大的计算与内存开销。本文提出一种全新设定，旨在通过一组为下游任务优化的共享参数，提升多种MLLMs的性能。为实现这一目标，我们提出可迁移视觉提示（Transferable Visual Prompting, TVP），这是一种简单有效的方法，仅需在单个模型上训练，生成的视觉提示即可迁移至不同模型并提升其在下游任务上的性能。我们引入两种策略，以解决现有视觉提示方法存在的跨模型特征损坏问题，增强所学提示的可迁移性：1）特征一致性对齐：对提示引发的特征变化施加约束，以保留任务无关知识；2）任务语义丰富化：借助语言引导，促使提示图像包含更丰富的任务特定语义。我们在6种现代MLLMs上开展了大量实验，涵盖目标识别、计数、多模态推理以及幻觉校正等多种任务，验证了TVP的有效性。
英文摘要：
Although Multimodal Large Language Models (MLLMs) have demonstrated promising versatile capabilities, their performance is still inferior to specialized models on downstream tasks, which makes adaptation necessary to enhance their utility. However, fine-tuning methods require independent training for every model, leading to huge computation and memory overheads. In this paper, we propose a novel setting where we aim to improve the performance of diverse MLLMs with a group of shared parameters optimized for a downstream task. To achieve this, we propose Transferable Visual Prompting (TVP), a simple and effective approach to generate visual prompts that can transfer to different models and improve their performance on downstream tasks after trained on only one model. We introduce two strategies to address the issue of cross-model feature corruption of existing visual prompting methods and enhance the transferability of the learned prompts, including 1) Feature Consistency Alignment: which imposes constraints to the prompted feature changes to maintain task-agnostic knowledge; 2) Task Semantics Enrichment: which encourages the prompted images to contain richer task-specific semantics with language guidance. We validate the effectiveness of TVP through extensive experiments with 6 modern MLLMs on a wide variety of tasks ranging from object recognition and counting to multimodal reasoning and hallucination correction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
