利用多模态项目反应理论评估跨模态推理能力与问题特征
Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory
AI总结：
M3IRT通过分解模型能力和项目难度，提供了一种评估多模态推理能力的框架，有效提升基准测试的可靠性和质量。
摘要：
多模态大语言模型（MLLMs）最近涌现出一种能够处理多种模态的通用架构。MLLMs的评估基准应衡量其跨模态整合能力。然而，当前的基准测试充满了捷径问题，这些问题仅需使用单一模态即可解决，从而导致排名不可靠。例如，在视觉-语言案例中，我们可以在没有图像或文本的情况下找到正确答案。这些低质量的问题不必要的增加了基准测试的规模和计算需求。我们引入了一个多模态和多维项目反应理论框架（M3IRT），它通过将模型能力与项目难度分解为仅图像、仅文本和跨模态组件来扩展经典IRT。M3IRT估计MLLMs的跨模态能力以及每个问题的跨模态难度，使能够生成紧凑、高质量的子集，更好地反映多模态推理。在24个VLMs上三个基准测试中，M3IRT优先选择真正的跨模态问题，即使当50%的项目被人为生成低质量问题时，也能保持排名的准确性，从而减少评估成本并提高可靠性。M3IRT因此提供了一种实用的工具，用于评估跨模态推理并改进多模态基准测试。
英文摘要：
Multimodal Large Language Models (MLLMs) have recently emerged as general architectures capable of reasoning over diverse modalities. Benchmarks for MLLMs should measure their ability for cross-modal integration. However, current benchmarks are filled with shortcut questions, which can be solved using only a single modality, thereby yielding unreliable rankings. For example, in vision-language cases, we can find the correct answer without either the image or the text. These low-quality questions unnecessarily increase the size and computational requirements of benchmarks. We introduce a multi-modal and multidimensional item response theory framework (M3IRT) that extends classical IRT by decomposing both model ability and item difficulty into image-only, text-only, and cross-modal components. M3IRT estimates cross-modal ability of MLLMs and each question's cross-modal difficulty, enabling compact, high-quality subsets that better reflect multimodal reasoning. Across 24 VLMs on three benchmarks, M3IRT prioritizes genuinely cross-modal questions over shortcuts and preserves ranking fidelity even when 50% of items are artificially generated low-quality questions, thereby reducing evaluation cost while improving reliability. M3IRT thus offers a practical tool for assessing cross-modal reasoning and refining multimodal benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
