PRISM-BENCH：超越最终答案准确率的基于视觉语言模型的 grounded 与忠实多模态推理诊断
PRISM-BENCH: Diagnosing Grounded and Faithful Multimodal Reasoning Beyond Final-Answer Accuracy
AI总结：
PRISM-BENCH基准通过专家整理的多模态推理问题及推理轨迹，提出感知探针、轨迹对齐和CORE指标，诊断视觉语言模型的 grounded 与忠实推理，发现前沿模型85%失败源于感知错误，最佳系统落后人类18.4个百分点。
摘要：
最终答案准确率仅报告视觉语言模型是否正确，却无法判断其答案是否基于图像、其陈述的推理是否反映该证据，以及生成答案的成本，而这些是决定系统能否部署的关键属性。我们提出 PRISM-BENCH，这是一个包含1000个专家整理问题的基准，涵盖5个代表性不足的推理类别，每个条目都带有一名人类标注者编写并经另一名验证的理想化推理轨迹。这些轨迹支持两项准确率无法提供的诊断：感知探针中，模型基于自身编写的描述作答，将前沿模型约85%的失败归因于感知错误，并将瓶颈置于 grounded；轨迹对齐可预测所有25个测试系统的答案正确性（宏观AUROC为0.66），且在长度控制下仍有效，因此测量的是推理内容而非冗长性。我们进一步引入 CORE，这是一个感知效率的指标，它在与响应长度不相关的同时，重新排序了24.2%的模型对相对于准确率的排名。在25个开放和封闭系统（包括GPT-5和Claude Opus 4.6）中，最佳准确率为66.0%；在4个有人类基准的类别中，最佳系统落后人类18.4个百分点，且近期前沿进展集中在识别而非空间 grounded。
英文摘要：
Final-answer accuracy reports whether a vision-language model was right, but not whether its answer was grounded in the image, whether its stated reasoning reflects that evidence, or what the answer cost to produce. Those are the properties that decide whether a system can be deployed. We present PRISM-BENCH, a benchmark of 1,000 expert-curated questions spanning five under-represented reasoning categories, in which every item carries an idealized reasoning trace written by one human annotator and validated by another. The traces support two diagnostics that accuracy cannot provide. A perception probe, in which the model answers from a description it wrote itself, attributes roughly 85% of frontier-model failures to misperception and places the bottleneck at grounding. Trace alignment predicts answer correctness for all 25 systems tested (macro AUROC 0.66) and survives length control, so it measures reasoning content and not verbosity. We further introduce CORE, an efficiency-aware metric that reorders 24.2% of model pairs relative to accuracy while remaining uncorrelated with response length. Across 25 open and closed systems, including GPT-5 and Claude Opus 4.6, the best accuracy is 66.0%. On the four categories with a human baseline the best system trails humans by 18.4 points, and recent frontier gains fall on recognition and not on spatial grounding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
