CRYSTAL基准通过可验证的中间步骤评估多模态推理,提出Match F1和Ordered Match F1两个指标,揭示模型在推理顺序、精度与召回率上的系统性缺陷,并引入CPR和CPR-Curriculum提升推理能力。
摘要:
我们介绍了CRYSTAL(通过生成步骤、可追溯性和逻辑的清晰推理)基准,包含6,372个实例,通过可验证的中间步骤评估多模态推理。我们提出了两个互补的指标:Match F1,通过语义相似性匹配评分步骤级精度和召回率,以及Ordered Match F1,进一步惩罚无序的推理链。参考通过一种受Delphi启发的流程生成,四个独立的MLLM生成轨迹,然后通过语义聚类聚合,并通过人工质量闸门验证。对20个MLLM的评估,包括在基准构建过程中未使用的商业前沿系统,揭示了在答案准确性之外的系统性失败:普遍的 cherry-picking(精度远高于召回率)、非单调的缩放权衡以及无序推理,其中没有竞争模型能保持超过60%的匹配步骤在正确顺序。除了评估外,我们提出因果过程奖励(CPR),一种乘法奖励,将答案正确性与步骤级对齐相结合,以及CPR-Curriculum,通过渐进增加推理难度进行训练。CPR-Curriculum通过GRPO实现32%的Match F1提升,优于加性奖励策略,提高推理能力而无需手动步骤标注。
英文摘要:
We introduce CRYSTAL (Clear Reasoning via Yielded Steps, Traceability, and Logic), a diagnostic benchmark with 6,372 instances that evaluates multimodal reasoning through verifiable intermediate steps. We propose two complementary metrics: Match F1, which scores step-level precision and recall via semantic similarity matching, and Ordered Match F1, which further penalizes disordered reasoning chains. References are constructed through a Delphi-inspired pipeline in which four independent MLLMs generate trajectories, which are then aggregated via semantic clustering and validated through human quality gates. Evaluation of 20 MLLMs, including commercial frontier systems not used during benchmark construction, reveals systematic failures that are invisible to answer accuracy: universal cherry-picking (precision far exceeds recall), non-monotonic scaling trade-offs, and disordered reasoning in which no competitive model preserves more than 60% of matched steps in the correct order. Beyond evaluation, we propose the Causal Process Reward (CPR), a multiplicative reward that couples answer correctness with step-level alignment, and CPR-Curriculum, which progressively increases reasoning difficulty during training. CPR-Curriculum achieves a 32% improvement in Match F1 via GRPO where additive reward strategies fail, improving reasoning without manual step annotation.
视觉问答(VQA)的主要方法依赖于使用“黑盒”神经编码器对图像和问题进行编码,并将单个令牌解码为“是”或“否”等答案。尽管这种方法有很强的定量结果,但它很难为预测过程提出直观的、人类可读的证明形式。为了解决这个不足,我们重新制定VQA作为一个完整的答案生成任务,这需要模型来证明其预测在自然语言。我们提出了LRTA [看,读,想,答],一个透明的神经符号推理框架的视觉问答,解决问题的一步一步像人类一样,并提供人类可读的形式的理由在每一步。具体来说,LRTA学习首先将图像转换为场景图,并将问题解析为多个推理指令。然后,它通过使用递归神经符号执行模块遍历场景图来一次执行一个推理指令。最后,它生成一个完整的答案,给定的问题与自然语言的理由。我们在GQA数据集上的实验表明,LRTA的性能大大优于最先进的模型(43.1% vs. 28.0%)的完整答案生成任务。我们还通过删除问题中的语言线索(属性和关系)来创建扰动GQA测试集,以分析模型是否具有表面数据相关性的智能猜测。我们表明,LRTA向真正理解这个问题迈出了一步,而最先进的模型往往从训练数据中学习表面的相关性。
mChartQA: A universal benchmark for multimodal Chart Question Answer based on Vision-Language Alignment and Reasoning
Shenyang Institute of Computing Technology, Chinese Academy of Sciences(中国科学院沈阳计算技术研究所);Beijing Wenge Technology Co., Ltd(北京文格科技有限公司);University of Chinese Academy of Sciences(中国科学院大学)
MuKEA: Multimodal Knowledge Extraction and Accumulation for Knowledge-based Visual Question Answering
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Université de Montréal(蒙特利尔大学);University of Adelaide(阿德莱德大学);Mila - Quebec AI Institute(米拉-魁北克人工智能研究所)
Beyond Retrieval: Joint Supervision and Multimodal Document Ranking for Textbook Question Answering
Faculty of Computing and Information Technology, King Abdulaziz University(计算机与信息科技学院,国王阿卜杜勒阿齐兹大学);College of Computer and Information Science, Imam Mohammad Ibn Saud Islamic University (IMSIU)(计算机与信息科学学院,伊玛目穆罕默德·本·萨乌德伊斯兰大学);School of Computing, Macquarie University(计算学院,麦考瑞大学)
FlowVQA: Mapping Multimodal Logic in Visual Question Answering with Flowcharts
Mercer Mettl(默瑟梅特尔);IGDTUW(英迪拉·甘地德里技术大学(女子));Google Research(谷歌研究院);Indian Institute of Technology Guwahati(古瓦哈蒂印度理工学院);University of Pennsylvania(宾夕法尼亚大学)
Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answering
Interdisciplinary Program in Artificial Intelligence(人工智能交叉学科项目);Department of Intelligence and Information(智能与信息系);Samsung Advanced Institute of Technology(三星先进技术研究院)
M$^3$-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所MAIS);Alibaba Cloud Computing(阿里巴巴云计算)
MARA: A Multimodal Adaptive Retrieval-Augmented Framework for Document Question Answering
Key Laboratory of Target Cognition and Application Technology (TCAT), AIRI, CAS(目标认知与应用技术重点实验室(TCAT),空气动力研究所,中国科学院);School of Electronic, Electrical and Communication Engineering, UCAS(电子、电气与通信工程学院,中国科学院大学);Aerospace Information Research Institute, Chinese Academy of Sciences(航天信息研究所,中国科学院);Baidu Inc.(百度公司)
MUTAN: Multimodal Tucker Fusion for Visual Question Answering
Sorbonne Universités, UPMC Univ Paris 06, CNRS, LIP6 UMR 7606(索邦大学,巴黎第六大学,法国国家科学研究中心,LIP6);Heuritech;Conservatoire National des Arts et Métiers(法国国立工艺学院)
Sorbonne Université(索邦大学);CNRS(法国国家科学研究中心);LIP6(巴黎第六大学计算机科学实验室);Carney Institute for Brain Science, Brown University(布朗大学卡尼脑科学研究所);Idiap Research Institute(伊迪亚普研究所);Australian Institute for Machine Learning, University of Adelaide(阿德莱德大学澳大利亚机器学习研究所)
Benchmarking Multimodal RAG through a Chart-based Document Question-Answering Generation Framework
College of Computer Science, Chongqing University(重庆大学计算机学院);Aerospace Information Research Institute, Chinese Academy of Sciences(中国科学院空天信息创新研究院);Kuaishou Technology(快手科技)