通过提示条件和头部选择解锁LVLMs的少样本能力
Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selection
AI总结：
本文提出Head Ensemble Classifiers (HEC)通过提示条件和头部选择提升LVLMs在少样本和零样本分类中的性能，实现与CLIP基方法的性能平衡。
摘要：
当前大型视觉语言模型（LVLMs）在许多零样本任务如图像描述、视觉问答和OCR中表现出色。然而，这些模型在图像分类任务上的表现较差，不如基于CLIP的方法。值得注意的是，这种差距令人惊讶，因为许多LVLMs使用CLIP预训练的视觉编码器。然而，LVLMs并不受CLIP架构的限制，具有独立的视觉和文本编码器。在CLIP中，这种分离使分类偏向于类名匹配而非联合视觉-文本推理。在本文中，我们展示了尽管LVLMs在原始性能上较差，但通过提示条件可以在推理过程中提高视觉特征的类别分离度，并且LVLMs的内部表示，尤其是注意力头，可以在零样本和少样本分类中超越模型本身。我们引入Head Ensemble Classifiers (HEC)来弥合基于CLIP和LVLM分类方法之间的性能差距。受高斯判别分析启发，HEC对最判别性的视觉和文本头部进行排名并组合成一个无需训练的分类器。我们证明HEC在12个数据集上的少样本和零样本分类中实现了最先进的性能。
英文摘要：
Current Large Vision Language Models (LVLMs) excel at many zero-shot tasks like image captioning, visual question answering and OCR. However, these same models suffer from poor performance at image classification tasks, underperforming against CLIP-based methods. Notably, this gap is surprising because many LVLMs use CLIP-pretrained vision encoders. Yet LVLMs are not inherently limited by CLIP's architecture with independent vision and text encoders. In CLIP, this separation biases classification toward class-name matching rather than joint visual-text reasoning. In this paper we show that, despite their poor raw performance, LVLMs can improve visual feature class separability at inference using prompt conditioning, and LVLMs' internal representations, especially attention heads, can outperform the model itself at zero-shot and few-shot classification. We introduce Head Ensemble Classifiers (HEC) to bridge the performance gap between CLIP-based and LVLM-based classification methods. Inspired by Gaussian Discriminant Analysis, HEC ranks the most discriminative vision and text heads and combines them into a training-free classifier. We show that HEC achieves state-of-the-art performance in few-shot and zero-shot classification across 12 datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
