arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.14542cs.AI
收藏

多模态基准真的有用吗?基于结构-响应协同校准的多模态基准项级诊断

Are Multimodal Benchmarks Really Useful? Item-Level Multimodal Benchmark Diagnosis via Structure-Response Co-Calibration

National University of Defense Technology(国防科技大学)

Shiqi Zhang, Weixin Zeng, Ziheng Zhang, Jiuyang Tang, Xiang Zhao

AI总结:

针对现有多模态基准存在捷径项、诊断方法不足的问题,提出SRCoD框架,通过结构-响应协同校准实现项级诊断,实验表明其性能优于现有最先进方法。

摘要:

多模态基准对于评估模型是否能对来自多个模态的信息进行推理至关重要。然而,现有研究表明,一些基准可能包含捷径项,这些项仅通过单一模态或答案选项的规律性即可回答,无法反映模型真实的跨模态推理能力。现有评估基准的方法通常依赖于内在项结构或外在模型响应行为,不足以进行全面诊断。为填补这一空白,我们提出SRCoD,一种用于多模态基准模态依赖项级诊断的结构-响应协同校准诊断框架。SRCoD从基准内容和目标答案中估计样本级多模态信息结构,并将该信号作为模态条件项响应建模的软结构锚。学习到的模型生成校准后的项级诊断轮廓,反映项对不同形式模态证据的依赖程度,支持超越整体准确率的细粒度基准分析。SRCoD还提供诊断结果的可解释归因。在两种互补协议(面向基准优化的验证和人工标注项验证)下开展的大量实验表明,SRCoD的性能优于最先进的基准诊断方法。

英文摘要:

Multimodal benchmarks are central to evaluating whether models can reason over information from multiple modalities. However, existing literature has revealed that some benchmarks may contain shortcut items that can be answered from a single modality or from answer-option regularities alone, thus failing to reflect models' true cross-modal inference ability. Existing methods for assessing benchmarks typically rely either on intrinsic item structure or extrinsic model response behavior, which are not adequate for a comprehensive diagnosis. To fill in this gap, we propose SRCoD, a structure-response co-calibrated diagnosis framework for item-level diagnosis of modality dependence in multimodal benchmarks. SRCoD estimates sample-wise multimodal information structure from benchmark content and target answers, and uses this signal as a soft structural anchor for modality-conditioned item response modeling. The learned model produces a calibrated item-level diagnostic profile that reflects how strongly an item depends on different forms of modality evidence, enabling fine-grained benchmark analysis beyond aggregate accuracy. SRCoD also provides interpretable attribution of the diagnosis results. Extensive experiments under two complementary protocols, i.e., benchmark-refinement oriented validation and human-labeled item validation, demonstrate that SRCoD outperforms state-of-the-art benchmark diagnosis methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑