PixFoundation: 我们在像素级视觉基础模型的方向正确吗？
PixFoundation: Are We Heading in the Right Direction with Pixel-level Vision Foundation Models?
AI总结：
PixFoundation提出新的基准测试，揭示像素级基础模型在视觉问答中的不足，并提出可解释性工具分析接地机制。
摘要：
近年来，多模态大语言模型（MLLMs）在像素级理解方面取得了显著进展。当前的趋势是通过在大规模标注数据上使用掩码进行像素级接地监督来训练MLLMs，并使用专门的解码器进行分割任务。然而，我们发现这些MLLMs在最近的以视觉为中心的基准测试中表现出在视觉问答（VQA）中的能力较弱。令人惊讶的是，一些方法甚至降低了从未使用像素级监督训练的MLLMs的接地能力。在本工作中，我们提出了两个新的具有配对评估的挑战性基准，用于VQA和接地。我们证明了简单的基线模型，即使不统一，也能达到与某些像素级MLLMs相匹配或超越的性能。我们的配对基准和评估使我们能够进一步分析VQA和/或接地失败的原因。此外，我们还提出了一种针对接地任务的语言和视觉提示的敏感性分析。更重要的是，我们研究了“当接地在MLLMs中出现时，相对于输出token时的时机？”这一研究问题。我们提出了一种可插拔的可解释性工具，可以插入任何MLLM中以研究上述问题。我们表明，接地不一定与输出中的精确指代表达一致，但可以与对象的部分、其位置、外观、上下文或状态一致。代码和数据集在https://msiam.github.io/PixFoundationSeries/上公开。
英文摘要：
Multiple works have emerged to push the boundaries of multi-modal large language models (MLLMs) towards pixel-level understanding. The current trend is to train MLLMs with pixel-level grounding supervision in terms of masks on large-scale labelled data and specialized decoders for the segmentation task. However, we show that such MLLMs when evaluated on recent challenging vision-centric benchmarks, exhibit a weak ability in visual question answering (VQA). Surprisingly, some of these methods even downgrade the grounding ability of MLLMs that were never trained with such pixel-level supervision. In this work, we propose two novel challenging benchmarks with paired evaluation for both VQA and grounding. We demonstrate that simple baselines that are not unified achieve performance that matches or surpasses some of the pixel-level MLLMs. Our paired benchmarks and evaluation enable additional analysis on the reasons for failure with respect to VQA and/or grounding. Furthermore, we propose a prompt sensitivity analysis on both the language and visual prompts tailored for the grounding task. More importantly, we study the research question of ``When does grounding emerge in MLLMs with respect to the output tokens?'' We propose an interpretability tool that can be plugged into any MLLM to study the aforementioned question. We show that grounding does not necessarily coincide with the exact referring expression in the output, but can coincide with the object parts, its location, appearance, context or state. Code and datasets are publicly available at https://msiam.github.io/PixFoundationSeries/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
