Prismatic VLMs：研究视觉条件语言模型的设计空间
Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models
AI总结：
本文系统研究了视觉条件语言模型的设计空间，编制标准化评估套件并沿关键设计轴进行严格分析，提出统一框架、优化代码和检查点，其7-13B规模模型严格优于现有最先进的开放VLMs。
摘要：
视觉条件语言模型（VLMs）在视觉对话、场景理解和机器人任务规划等应用中的采用日益增多；这种采用推动了大量新模型的涌现，如 LLaVa、InstructBLIP 和 PaLI-3。尽管新发布数量庞大，但围绕图像预处理、架构和优化的关键设计决策仍探索不足，这使得理解哪些因素决定了模型性能变得具有挑战性——缺乏客观、一致的评估进一步加剧了这一挑战。为了解决这些差距，我们首先编制了一套标准化评估套件，涵盖视觉问答、物体定位以及探测幻觉等属性的挑战集；这些评估提供了对 VLM 能力的细粒度洞察。其次，我们严格沿着关键设计轴研究 VLMs，包括预训练视觉表示以及从基础语言模型与指令微调语言模型进行训练等。我们将分析与三项资源贡献相结合：（1）用于评估 VLMs 的统一框架，（2）优化且灵活的训练代码，（3）所有模型的检查点，包括一系列 7-13B 规模的 VLMs，其性能严格优于开放 VLM 领域最先进的 InstructBLIP 和 LLaVa v1.5。
英文摘要：
Visually-conditioned language models (VLMs) have seen growing adoption in applications such as visual dialogue, scene understanding, and robotic task planning; adoption that has fueled a wealth of new models such as LLaVa, InstructBLIP, and PaLI-3. Despite the volume of new releases, key design decisions around image preprocessing, architecture, and optimization are under-explored, making it challenging to understand what factors account for model performance $-$ a challenge further complicated by the lack of objective, consistent evaluations. To address these gaps, we first compile a suite of standardized evaluations spanning visual question answering, object localization, and challenge sets that probe properties such as hallucination; evaluations that provide fine-grained insight VLM capabilities. Second, we rigorously investigate VLMs along key design axes, including pretrained visual representations and training from base vs. instruct-tuned language models, amongst others. We couple our analysis with three resource contributions: (1) a unified framework for evaluating VLMs, (2) optimized, flexible training code, and (3) checkpoints for all models, including a family of VLMs at the 7-13B scale that strictly outperform InstructBLIP and LLaVa v1.5, the state-of-the-art in open VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
