AMVICC: 一种用于VLM和IGM跨模态故障模式分析的新型基准
AMVICC: A Novel Benchmark for Cross-Modal Failure Mode Profiling for VLMs and IGMs
AI总结:
提出AMVICC基准,通过图像到文本和文本到图像任务系统比较多模态大模型和图像生成模型的视觉推理失败模式,发现故障模式在模型和模态间共享,但存在特定于模型和模态的失败。
摘要:
我们通过创建一个新型基准,系统比较图像到文本和文本到图像任务中的失败模式,实现视觉理解的跨模态评估,从而研究多模态大语言模型(MLLMs)和图像生成模型(IGMs)的视觉推理局限性。尽管机器学习快速发展,视觉语言模型(VLMs)仍然无法理解基本视觉概念,如物体方向、数量和空间关系,这突显了基础视觉推理的差距。通过将MMVP基准问题改编为显式和隐式提示,我们创建了\textit{AMVICC},一个用于分析跨模态失败模式的新型基准。在测试了9个视觉推理类别中的11个MLLMs和3个IGMs后,我们的结果表明,失败模式通常在模型和模态之间共享。然而,某些失败是模型特定和模态特定的,这可能归因于各种因素。IGMs在响应提示时始终难以操纵特定的视觉组件,尤其是在显式提示中,表明对细粒度视觉属性的控制能力较差。我们的发现最直接适用于现有最先进模型在结构化视觉推理任务上的评估。这项工作为未来的跨模态对齐研究奠定了基础,提供了一个框架来探究图像生成和视觉解释失败是否源于共同的局限性。这些见解可以指导未来统一视觉语言建模的改进。
英文摘要:
We investigate visual reasoning limitations of both multimodal large language models (MLLMs) and image generation models (IGMs) by creating a novel benchmark to systematically compare failure modes across image-to-text and text-to-image tasks, enabling cross-modal evaluation of visual understanding. Despite rapid growth in machine learning, vision language models (VLMs) still fail to understand basic visual concepts such as object orientation, quantity, and spatial relationships, which highlights gaps in elementary visual reasoning. By adapting MMVP benchmark questions into explicit and implicit prompts, we create \textit{AMVICC}, a novel benchmark for profiling failure modes across various modalities. After testing 11 MLLMs and 3 IGMs in 9 categories of visual reasoning, our results show that failure modes are often shared between models and modalities. However, certain failures are model-specific and modality-specific, and this can potentially be attributed to various factors. IGMs consistently struggle to manipulate specific visual components in response to prompts, especially in explicit prompts, suggesting poor control over fine-grained visual attributes. Our findings apply most directly to the evaluation of existing state-of-the-art models on structured visual reasoning tasks. This work lays the foundation for future cross-modal alignment studies, offering a framework to probe whether image generation and visual interpretation failures stem from shared limitations. These insights can guide future improvements in unified vision-language modeling.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。