大型生成式视觉-语言模型组合性的考察
An Examination of the Compositionality of Large Generative Vision-Language Models
AI总结：
本文针对生成式视觉-语言模型组合性评估中的句法偏差问题，提出SyntaxBias Score和新任务，并构建无偏基准SADE，为多模态组合推理研究提供更可靠的评测工具。
摘要：
随着大型语言模型（LLMs）的成功，许多生成式视觉-语言模型（GVLMs）通过多模态指令微调被构建出来。然而，GVLMs在多模态组合推理方面的性能仍未得到充分探索。在本文中，我们考察了用于评估GVLMs组合性的评估指标（VisualGPTScore等）和现有基准。我们识别出当前基准中的句法偏差，该偏差被GVLMs的语言能力所利用。这种偏差使得VisualGPTScore成为评估GVLMs的不充分指标。为应对这一问题，我们首先引入SyntaxBias Score，利用LLMs量化此类偏差以进行缓解。随后添加了一个具有挑战性的新任务，用于评估GVLMs对句法正确性固有倾向的鲁棒性。使用偏差缓解后的数据集和新任务，我们提出了一个新的基准，即SyntActically DE-biased benchmark（SADE）。我们的研究为GVLMs的组合性提供了一个无偏基准，有助于该方向的未来研究（代码和数据集可在https://github.com/TeleeMa/SADE获取）。
英文摘要：
With the success of Large Language Models (LLMs), many Generative Vision-Language Models (GVLMs) have been constructed via multimodal instruction tuning. However, the performance of GVLMs in multimodal compositional reasoning remains under-explored. In this paper, we examine both the evaluation metrics (VisualGPTScore, etc.) and current benchmarks for evaluating the compositionality of GVLMs. We identify the syntactical bias in current benchmarks, which is exploited by the linguistic capability of GVLMs. The bias renders VisualGPTScore an insufficient metric for assessing GVLMs. To combat this, we first introduce a SyntaxBias Score, leveraging LLMs to quantify such bias for mitigation. A challenging new task is subsequently added to evaluate the robustness of GVLMs against inherent inclination toward syntactical correctness. Using the bias-mitigated datasets and the new task, we propose a novel benchmark, namely SyntActically DE-biased benchmark (SADE). Our study provides an unbiased benchmark for the compositionality of GVLMs, facilitating future research in this direction (Code and dataset are available at https://github.com/TeleeMa/SADE).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
