arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.13622cs.CVcs.AI
收藏

CARPE:通过集成实现上下文感知的图像表示优先级

CARPE: Context-Aware Image Representation Prioritization via Ensemble for Large Vision-Language Models

University of California, Davis(加州大学戴维斯分校)

Donghee Lee, Rui Cai, Zhe Zhao

AI总结:

CARPE通过集成机制增强大视觉-语言模型对视觉和文本模态的自适应加权能力,提升图像分类和多模态任务表现。

摘要:

大型视觉-语言模型(LVLMs)通常通过自回归语言建模目标进行训练,将视觉表示对齐到语言空间。尽管在多模态推理中有效,这种对齐可能削弱视觉中心能力,导致LVLMs在图像分类等任务上表现不佳。为此,我们提出了一种轻量级框架CARPE,通过视觉集成层和上下文感知集成机制,将原始视觉特征与对齐的LLM表示结合。该设计增强了模型自适应加权视觉和文本模态的能力,并使模型能够捕捉图像表示的多个方面。大量实验表明,CARPE在图像分类和多样化的视觉-语言基准上均提升了性能。我们的结果表明,模态平衡在多模态泛化中起关键作用,通过改进自回归LVLMs中的表示利用。

英文摘要:

Large vision-language models (LVLMs) are typically trained using autoregressive language modeling objectives, which align visual representations with linguistic space. While effective for multimodal reasoning, this alignment can weaken vision-centric capabilities, causing LVLMs to underperform their base vision encoders on tasks such as image classification. To address this limitation, we propose Context-Aware Image Representation Prioritization via Ensemble (CARPE), a lightweight framework that integrates raw vision features with aligned LLM representations through vision-integration layers and a context-aware ensemble mechanism. This design enhances the model's ability to adaptively weight visual and textual modalities and enables the model to capture various aspects of image representations. Extensive experiments demonstrate that CARPE improves performance on both image classification and diverse vision-language benchmarks. Our results suggest that modality balancing plays a critical role in multimodal generalization by improving representation utilization within autoregressive LVLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑