arXiv 2606.190532026-06-18cs.CV

大规模视觉-语言模型在细粒度图像任务上的基准测试：从评估到诊断

Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis

School of Computer Science and Engineering, Southeast University, China（东南大学计算机科学与工程学院，中国）；Alibaba Group（阿里巴巴集团）；School of Computer Science and Engineering, School of Intelligence Science and Engineering, and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, China（东南大学计算机科学与工程学院、智能科学与工程学院以及新一代人工智能技术及其交叉应用关键实验室，中国）；Wangxuan Institute of Computer Technology, National Key Laboratory for Multimedia Information Processing, Peking University, China（北京大学王轩计算机技术研究所、多媒体信息处理国家重点实验室，中国）；University of Copenhagen, Denmark（丹麦哥本哈根大学）

Hong-Tao Yu, Chen-Wei Xie, Yuxin Peng, Serge Belongie, Xiu-Shen Wei

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出FG-BMK基准，含101万问题和28万图像，通过人机双范式评估LVLM的细粒度语义识别与视觉判别能力，诊断失败原因，发现视觉表示、语义对齐等瓶颈。

摘要：

近期大规模视觉-语言模型（LVLMs）展示了显著的多模态感知和推理能力。尽管众多基准从整体或任务特定角度评估了LVLMs，但它们在细粒度图像任务（计算机视觉的基础）上的能力仍未得到充分理解。为填补这一空白，我们引入FG-BMK，一个全面的细粒度评估基准，包含101万问题和28万图像，覆盖从常见物体中心领域到专业领域的多样化场景。FG-BMK通过面向人类和面向机器的范式，联合评估对话级细粒度语义识别和特征级视觉判别能力，从而诊断分析LVLM的失败是否源于视觉表示不足、视觉-语义对齐薄弱或细粒度知识有限。通过对一系列代表性LVLM/VLM的大量实验，我们发现当前LVLMs仍是不充分的细粒度识别器，失败源于视觉表示、语义对齐、模态对齐和类别级知识中相互交织的瓶颈。我们进一步分析了提升细粒度能力的训练设计因素，并考察了视觉和语言扰动如何影响LVLM预测。这些发现为当前LVLMs的局限性提供了诊断性见解，并为未来数据构建和模型设计提供了指导，以开发更可靠的细粒度视觉任务LVLMs。我们的代码已开源，可从此https URL获取。

英文摘要：

Recent advancements in Large Vision-Language Models (LVLMs) have demonstrated remarkable multimodal perception and reasoning capabilities. While numerous benchmarks have evaluated LVLMs from holistic or task-specific perspectives, their capabilities on fine-grained image tasks-fundamental to computer vision-remain insufficiently understood. To address this gap, we introduce FG-BMK, a comprehensive fine-grained evaluation benchmark containing 1.01 million questions and 0.28 million images, covering diverse scenarios from common object-centric domains to specialized domains. FG-BMK jointly evaluates dialogue-level fine-grained semantic recognition and feature-level visual discriminability through human-oriented and machine-oriented paradigms, enabling diagnostic analysis of whether LVLM failures arise from insufficient visual representations, weak visual-to-semantic grounding, or limited fine-grained knowledge. Through extensive experiments on a diverse set of representative LVLMs/VLMs, we find that current LVLMs remain inadequate fine-grained recognizers, with failures arising from intertwined bottlenecks in visual representations, semantic grounding, modality alignment, and category-level knowledge. We further analyze training design factors for improving fine-grained capabilities and examine how visual and linguistic perturbations affect LVLM predictions. These findings provide diagnostic insights into the limitations of current LVLMs and offer guidance for future data construction and model design in developing more reliable LVLMs for fine-grained visual tasks. Our code is open-source and available at https://fg-bmk.github.io/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。