BRAVE：拓宽视觉语言模型的视觉编码
BRAVE: Broadening the visual encoding of vision-language models
AI总结：
本文提出 BRAVE，通过整合多个冻结视觉编码器的特征形成通用压缩表示并输入冻结语言模型，在图像描述与 VQA 基准上以更少可训练参数达到 SOTA。
摘要：
视觉语言模型（VLM）通常由视觉编码器（例如 CLIP）和语言模型（LM）组成，后者解释编码后的特征以解决下游任务。尽管取得了显著进展，由于视觉编码器能力有限，VLM 仍存在若干缺陷，例如对某些图像特征“视而不见”、视觉幻觉等。为解决这些问题，我们研究拓宽 VLM 的视觉编码能力。我们首先全面基准测试了若干具有不同归纳偏置、用于解决 VLM 任务的视觉编码器。我们观察到，不存在单一编码配置能够在不同任务上持续取得最佳性能，而具有不同偏置的编码器可能表现得出奇地相似。受此启发，我们提出一种名为 BRAVE 的方法，将来自多个冻结编码器的特征整合为一种更通用的表示，并可直接作为输入提供给冻结的 LM。BRAVE 在广泛的图像描述和 VQA 基准上取得了最先进性能，并显著缓解了上述 VLM 问题，同时与现有方法相比需要更少的可训练参数，并具有更压缩的表示。我们的结果凸显了融合不同视觉偏置的潜力，可使 VLM 获得更广泛、更具情境化的视觉理解。
英文摘要：
Vision-language models (VLMs) are typically composed of a vision encoder, e.g. CLIP, and a language model (LM) that interprets the encoded features to solve downstream tasks. Despite remarkable progress, VLMs are subject to several shortcomings due to the limited capabilities of vision encoders, e.g. "blindness" to certain image features, visual hallucination, etc. To address these issues, we study broadening the visual encoding capabilities of VLMs. We first comprehensively benchmark several vision encoders with different inductive biases for solving VLM tasks. We observe that there is no single encoding configuration that consistently achieves top performance across different tasks, and encoders with different biases can perform surprisingly similarly. Motivated by this, we introduce a method, named BRAVE, that consolidates features from multiple frozen encoders into a more versatile representation that can be directly fed as the input to a frozen LM. BRAVE achieves state-of-the-art performance on a broad range of captioning and VQA benchmarks and significantly reduces the aforementioned issues of VLMs, while requiring a smaller number of trainable parameters than existing methods and having a more compressed representation. Our results highlight the potential of incorporating different visual biases for a more broad and contextualized visual understanding of VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
