Semantic-Clipping：基于语义引导视觉选择的高效视觉-语言建模
Semantic-Clipping: Efficient Vision-Language Modeling with Semantic-Guidedd Visual Selection
AI总结：
本文提出 SEMCLIP 框架，利用文本语义引导选择关键视觉区域并融入视觉编码，无需重训即可提升 VLM 细粒度理解与效率。
摘要：
视觉-语言模型（VLM）利用对齐的视觉编码器将图像转换为视觉 token，使其能够像文本一样由骨干大语言模型（LLM）处理。这种统一输入范式使 VLM 在视觉问答（VQA）等视觉-语言任务中表现出色。为提升细粒度视觉推理能力，近期视觉-语言建模进展引入图像裁剪技术，将所有编码后的子图像输入模型。然而，该方法显著增加视觉 token 数量，导致效率低下，并可能分散 LLM 的注意力。为解决 VLM 中图像表示的泛化挑战，我们提出一个轻量级、通用框架，可无缝集成到现有 VLM 中，以增强其处理细粒度细节的能力。我们的方法利用文本语义识别关键视觉区域，无需对 VLM 进行任何再训练即可提升 VQA 性能。此外，该方法将文本信号纳入视觉编码过程，同时提高效率和效果。所提出的方法 SEMCLIP 在 7 个基准上使 7B VLM LLaVA-1.5 的视觉理解能力平均增强 3.3%，在具有挑战性的细节理解基准 V* 上尤其提升 5.3%。
英文摘要：
Vision-Language Models (VLMs) leverage aligned visual encoders to transform images into visual tokens, allowing them to be processed similarly to text by the backbone large language model (LLM). This unified input paradigm enables VLMs to excel in vision-language tasks such as visual question answering (VQA). To improve fine-grained visual reasoning, recent advancements in vision-language modeling introduce image cropping techniques that feed all encoded sub-images into the model. However, this approach significantly increases the number of visual tokens, leading to inefficiency and potential distractions for the LLM. To address the generalization challenges of image representation in VLMs, we propose a lightweight, universal framework that seamlessly integrates with existing VLMs to enhance their ability to process finegrained details. Our method leverages textual semantics to identify key visual areas, improving VQA performance without requiring any retraining of the VLM. Additionally, it incorporates textual signals into the visual encoding process, enhancing both efficiency and effectiveness. The proposed method, SEMCLIP, strengthens the visual understanding of a 7B VLM, LLaVA-1.5 by 3.3% on average across 7 benchmarks, and particularly by 5.3% on the challenging detailed understanding benchmark V*.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
