arXiv 2404.123902024-07-03cs.CVcs.AIcs.CL

BLINK：多模态大语言模型能看见但无法感知

BLINK: Multimodal Large Language Models Can See but Not Perceive

University of Pennsylvania（宾夕法尼亚大学）；University of Washington（华盛顿大学）；Allen Institute for AI（艾伦人工智能研究所）；University of California, Davis（加利福尼亚大学戴维斯分校）；Columbia University（哥伦比亚大学）

Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, Ranjay Krishna

备注：: Multimodal Benchmark, Project Url: https://zeyofu.github.io/blink/, ECCV 2024

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出 Blink 基准，将 14 个经典视觉感知任务转化为 3807 道多选题，揭示现有多模态大模型远未达到人类水平，并指出专用 CV 模型的优势可为未来改进提供方向。

摘要：

我们提出了 Blink，一个针对多模态语言模型（LLMs）的新基准，专注于其他评估中不存在的核心视觉感知能力。Blink 中的大多数任务可以被人类“一眨眼间”解决（例如相对深度估计、视觉对应、取证检测和多视角推理）。然而，我们发现这些需要感知的任务对当前的多模态 LLMs 构成了重大挑战，因为它们难以通过自然语言进行中介。Blink 将 14 个经典计算机视觉任务重新格式化为 3,807 道多项选择题，并配有单张或多张图像以及视觉提示。人类平均准确率达到 95.70%，而 Blink 对现有多模态 LLMs 而言出人意料地困难：即使表现最好的 GPT-4V 和 Gemini 也仅分别达到 51.26% 和 45.72% 的准确率，仅比随机猜测高出 13.17% 和 7.63%，表明这些感知能力在近期的多模态 LLMs 中尚未“涌现”。我们的分析还强调，专用 CV 模型能够更好地解决这些问题，这为未来的改进指明了潜在路径。我们相信 Blink 将推动社区帮助多模态 LLMs 赶上人类水平的视觉感知。

英文摘要：

We introduce Blink, a new benchmark for multimodal language models (LLMs) that focuses on core visual perception abilities not found in other evaluations. Most of the Blink tasks can be solved by humans "within a blink" (e.g., relative depth estimation, visual correspondence, forensics detection, and multi-view reasoning). However, we find these perception-demanding tasks cast significant challenges for current multimodal LLMs because they resist mediation through natural language. Blink reformats 14 classic computer vision tasks into 3,807 multiple-choice questions, paired with single or multiple images and visual prompting. While humans get 95.70% accuracy on average, Blink is surprisingly challenging for existing multimodal LLMs: even the best-performing GPT-4V and Gemini achieve accuracies of 51.26% and 45.72%, only 13.17% and 7.63% higher than random guessing, indicating that such perception abilities have not "emerged" yet in recent multimodal LLMs. Our analysis also highlights that specialist CV models could solve these problems much better, suggesting potential pathways for future improvements. We believe Blink will stimulate the community to help multimodal LLMs catch up with human-level visual perception.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。