揭秘无编码器视觉-语言模型
Unveiling Encoder-Free Vision-Language Models
AI总结：
针对现有视觉-语言模型依赖视觉编码器带来的归纳偏置问题，该研究提出无编码器模型EVE及高效训练策略，仅用35M公开数据便在多基准上媲美同规模有编码器模型，性能优于Fuyu-8B，为纯解码器跨模态架构提供透明高效方案。
摘要：
现有视觉-语言模型（VLM）大多依赖视觉编码器提取视觉特征，再结合大语言模型（LLM）完成视觉-语言任务。然而，视觉编码器在抽象视觉表征时会引入强归纳偏置，例如分辨率、宽高比和语义先验，这可能会阻碍VLM的灵活性与效率。训练可接收无缝视觉与语言输入的纯VLM（即无视觉编码器的模型）仍具挑战性，且相关探索较少。实证观察表明，无编码器的直接训练会导致收敛缓慢且性能差距较大。\n在本研究中，我们填补了基于编码器模型与无编码器模型之间的空白，提出了一种简单却有效的纯VLM训练方案。具体而言，我们通过全面实验揭示了高效训练无编码器VLM的关键要点：（1）在统一的解码器内桥接视觉-语言表征；（2）通过额外监督增强视觉识别能力。基于这些策略，我们推出了EVE——一种可高效训练与前向推理的无编码器视觉-语言模型。值得注意的是，仅使用3500万公开可访问数据，EVE就能在多个视觉-语言基准上与同等规模的基于编码器的VLM相媲美，其性能显著优于训练流程神秘、训练数据未公开的同类模型Fuyu-8B。我们认为EVE为开发跨模态的纯解码器架构提供了一条透明且高效的路径。我们的代码与模型已公开发布于：https://github.com/baaivision/EVE。
英文摘要：
Existing vision-language models (VLMs) mostly rely on vision encoders to extract visual features followed by large language models (LLMs) for visual-language tasks. However, the vision encoders set a strong inductive bias in abstracting visual representation, e.g., resolution, aspect ratio, and semantic priors, which could impede the flexibility and efficiency of the VLMs. Training pure VLMs that accept the seamless vision and language inputs, i.e., without vision encoders, remains challenging and rarely explored. Empirical observations reveal that direct training without encoders results in slow convergence and large performance gaps. In this work, we bridge the gap between encoder-based and encoder-free models, and present a simple yet effective training recipe towards pure VLMs. Specifically, we unveil the key aspects of training encoder-free VLMs efficiently via thorough experiments: (1) Bridging vision-language representation inside one unified decoder; (2) Enhancing visual recognition capability via extra supervision. With these strategies, we launch EVE, an encoder-free vision-language model that can be trained and forwarded efficiently. Notably, solely utilizing 35M publicly accessible data, EVE can impressively rival the encoder-based VLMs of similar capacities across multiple vision-language benchmarks. It significantly outperforms the counterpart Fuyu-8B with mysterious training procedures and undisclosed training data. We believe that EVE provides a transparent and efficient route for developing a pure decoder-only architecture across modalities. Our code and models are publicly available at: https://github.com/baaivision/EVE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
