arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.00142cs.CVcs.AIcs.CL
收藏

利用大型多模态模型特征增强少样本视觉-语言分类

Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Features

Carnegie Mellon University(卡内基梅隆大学);University of California, Berkeley(加州大学伯克利分校);IBM Research(IBM研究院);MIT-IBM Watson AI Lab(麻省理工-IBM沃森人工智能实验室)

Chancharik Mitra, Brandon Huang, Tianning Chai, Zhiqiu Lin, Assaf Arbelle, Rogerio Feris, Leonid Karlinsky, Trevor Darrell, Deva Ramanan, Roei Herzig

AI总结:

本文提出无需微调的稀疏注意力向量 SAV,从生成式 LMM 的少于 5% 注意力头激活中提取特征,在少样本视觉-语言分类任务上达到先进性能并具备可扩展性与泛化性。

摘要:

诸如 LLaVA 和 Qwen-VL 之类的生成式大型多模态模型(LMM)擅长多种视觉-语言(VL)任务。尽管性能强劲,LMM 的生成式输出并不专门适用于视觉-语言分类任务(即具有视觉-语言输入和离散标签的任务),例如图像分类和多选 VQA。将 LMM 用于这些任务的一个关键挑战,是从生成式 LMM 中提取有用特征。为克服这一问题,我们提出一种利用从 LMM 潜在空间进行多模态特征提取的方法。为此,我们提出了稀疏注意力向量(SAV)——一种无需微调的方法,它利用 LMM 中稀疏的注意力头激活(少于 5% 的注意力头)作为强特征表示。仅使用少样本示例,SAV 在一系列视觉-语言分类任务上,与多种少样本和经过微调的基线相比展现出最先进的性能。我们的实验还表明,SAV 的性能可随额外示例增加而扩展,并能泛化到类似任务,从而确立 SAV 是有效且鲁棒的多模态特征表示。

英文摘要:

Generative Large Multimodal Models (LMMs) like LLaVA and Qwen-VL excel at a wide variety of vision-language (VL) tasks. Despite strong performance, LMMs' generative outputs are not specialized for vision-language classification tasks (i.e., tasks with vision-language inputs and discrete labels) such as image classification and multiple-choice VQA. One key challenge in utilizing LMMs for these tasks is the extraction of useful features from generative LMMs. To overcome this, we propose an approach that leverages multimodal feature extraction from the LMM's latent space. Toward this end, we present Sparse Attention Vectors (SAVs) -- a finetuning-free method that leverages sparse attention head activations (fewer than 5% of the heads) in LMMs as strong feature representations. With only few-shot examples, SAVs demonstrate state-of-the-art performance compared to a variety of few-shot and finetuned baselines on a collection of vision-language classification tasks. Our experiments also imply that SAVs can scale in performance with additional examples and generalize to similar tasks, establishing SAVs as both effective and robust multimodal feature representations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑