arXiv 2308.065952023-12-26cs.CLcs.AIcs.CV

VisIT-Bench：受真实世界应用启发的视觉语言指令遵循基准

VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use

Hebrew University（希伯来大学）；Google Research（谷歌研究院）；UCLA（加州大学洛杉矶分校）；Allen Institute for AI（艾伦人工智能研究所）；University of Washington（华盛顿大学）；UCSB（加州大学圣塔芭芭拉分校）；Stanford（斯坦福大学）；LAION

Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, Ludwig Schmidt

备注：: Accepted to NeurIPS 2023, Datasets and Benchmarks. Website: https://visit-bench.github.io/

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

VisIT-Bench是一个包含70个指令族、592个测试查询的视觉语言指令遵循基准，通过人工验证参考和文本LLM自动评估，揭示模型与GPT-4参考间存在显著质量差距。

摘要：

我们推出了VisIT-Bench（视觉指令基准），这是一个用于评估面向真实世界应用的指令遵循视觉语言模型的基准。我们的出发点是为指令调优的视觉语言模型应该能够处理的70个“指令族”进行策划。在VQAv2和COCO等评估的基础上，任务范围从基础识别到游戏玩法和创意生成。经过策划，我们的数据集包含592个测试查询，每个查询都带有人工撰写的指令条件化描述。这些描述揭示了指令特定的因素，例如，对于询问轮椅使用者商店入口可达性的指令，指令条件化描述会描述坡道/潜在障碍。这些描述使得能够1）为每个实例收集人工验证的参考输出；2）使用仅文本的LLM自动评估候选多模态生成，与人类判断保持一致。我们使用人工和自动评估来量化模型与参考之间的质量差距；例如，表现最好的指令遵循模型在与GPT-4参考的比较中仅赢得27%的对比。VisIT-Bench的参与是动态的，从业者只需在项目网站上提交其模型的响应；数据、代码和排行榜可在visit-bench.github.io获取。

英文摘要：

We introduce VisIT-Bench (Visual InsTruction Benchmark), a benchmark for evaluation of instruction-following vision-language models for real-world use. Our starting point is curating 70 'instruction families' that we envision instruction tuned vision-language models should be able to address. Extending beyond evaluations like VQAv2 and COCO, tasks range from basic recognition to game playing and creative generation. Following curation, our dataset comprises 592 test queries, each with a human-authored instruction-conditioned caption. These descriptions surface instruction-specific factors, e.g., for an instruction asking about the accessibility of a storefront for wheelchair users, the instruction-conditioned caption describes ramps/potential obstacles. These descriptions enable 1) collecting human-verified reference outputs for each instance; and 2) automatic evaluation of candidate multimodal generations using a text-only LLM, aligning with human judgment. We quantify quality gaps between models and references using both human and automatic evaluations; e.g., the top-performing instruction-following model wins against the GPT-4 reference in just 27% of the comparison. VisIT-Bench is dynamic to participate, practitioners simply submit their model's response on the project website; Data, code and leaderboard is available at visit-bench.github.io.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。