arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.13099cs.AIcs.CVcs.IRcs.MM
收藏

超越最终答案:CRYSTAL基准用于透明多模态推理评估

Beyond Final Answers: CRYSTAL Benchmark for Transparent Multimodal Reasoning Evaluation

Wayner Barrios, SouYoung Jin

AI总结:

CRYSTAL基准通过可验证的中间步骤评估多模态推理,提出Match F1和Ordered Match F1两个指标,揭示模型在推理顺序、精度与召回率上的系统性缺陷,并引入CPR和CPR-Curriculum提升推理能力。

摘要:

我们介绍了CRYSTAL(通过生成步骤、可追溯性和逻辑的清晰推理)基准,包含6,372个实例,通过可验证的中间步骤评估多模态推理。我们提出了两个互补的指标:Match F1,通过语义相似性匹配评分步骤级精度和召回率,以及Ordered Match F1,进一步惩罚无序的推理链。参考通过一种受Delphi启发的流程生成,四个独立的MLLM生成轨迹,然后通过语义聚类聚合,并通过人工质量闸门验证。对20个MLLM的评估,包括在基准构建过程中未使用的商业前沿系统,揭示了在答案准确性之外的系统性失败:普遍的 cherry-picking(精度远高于召回率)、非单调的缩放权衡以及无序推理,其中没有竞争模型能保持超过60%的匹配步骤在正确顺序。除了评估外,我们提出因果过程奖励(CPR),一种乘法奖励,将答案正确性与步骤级对齐相结合,以及CPR-Curriculum,通过渐进增加推理难度进行训练。CPR-Curriculum通过GRPO实现32%的Match F1提升,优于加性奖励策略,提高推理能力而无需手动步骤标注。

英文摘要:

We introduce CRYSTAL (Clear Reasoning via Yielded Steps, Traceability, and Logic), a diagnostic benchmark with 6,372 instances that evaluates multimodal reasoning through verifiable intermediate steps. We propose two complementary metrics: Match F1, which scores step-level precision and recall via semantic similarity matching, and Ordered Match F1, which further penalizes disordered reasoning chains. References are constructed through a Delphi-inspired pipeline in which four independent MLLMs generate trajectories, which are then aggregated via semantic clustering and validated through human quality gates. Evaluation of 20 MLLMs, including commercial frontier systems not used during benchmark construction, reveals systematic failures that are invisible to answer accuracy: universal cherry-picking (precision far exceeds recall), non-monotonic scaling trade-offs, and disordered reasoning in which no competitive model preserves more than 60% of matched steps in the correct order. Beyond evaluation, we propose the Causal Process Reward (CPR), a multiplicative reward that couples answer correctness with step-level alignment, and CPR-Curriculum, which progressively increases reasoning difficulty during training. CPR-Curriculum achieves a 32% improvement in Match F1 via GRPO where additive reward strategies fail, improving reasoning without manual step annotation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑