CulMind：中国文化遗产中的多模态理解与推理基准
CulMind: Benchmarking Multimodal Understanding and Reasoning in Chinese Cultural Heritage
AI总结：
针对现有基准忽视推理过程的问题，提出CulMind和CulMind-R基准，涵盖50个任务和24个推理子任务，并设计ReaScore指标评估推理质量，揭示答案与推理之间的差距。
摘要：
评估多模态大语言模型（MLLMs）在中国文化遗产（CCH）中的表现需要对视觉、文本、风格和历史线索进行细粒度推理。然而，现有的CCH基准主要强调最终答案的准确性，而推理过程的准确性和完整性仍未得到充分探索。为弥补这一空白，我们引入了CulMind和CulMind-R：一个高质量的多模态CCH基准，涵盖来自100多家博物馆藏品的50个任务，以及一个24个任务的推理子集，该子集自适应地定义了用于推理过程评估的任务特定维度。为了评估推理质量，我们提出了ReaScore，一种任务自适应指标，通过自动加权任务相关维度来评估推理。对14个领先的MLLMs的实验揭示了答案与推理之间的显著差距，尤其是在具有挑战性的任务上。进一步分析表明，任务自适应的维度选择和加权能更好地使评估结果与专家判断对齐。总体而言，我们的基准和指标支持更符合专家标准的CCH理解评估，并为更广泛的文化遗产评估提供了可迁移的参考。我们公开发布数据、代码和评估脚本于https://this URL，以促进可重复研究。
英文摘要：
Evaluating Multimodal Large Language Models (MLLMs) in Chinese Cultural Heritage (CCH) requires fine-grained reasoning over visual, textual, stylistic, and historical clues. However, existing CCH benchmarks mainly emphasize final-answer accuracy, while the accuracy and completeness of reasoning processes remain underexplored. To address this gap, we introduce CulMind and CulMind-R: a high-quality benchmark for multimodal CCH covering 50 tasks from collections of more than 100 museums, and a 24-task reasoning subset that adaptively defines task-specific dimensions for reasoning process evaluation. To evaluate reasoning quality, we propose ReaScore, a task-adaptive metric that evaluates reasoning by automatically weighting task-relevant dimensions. Experiments on 14 leading MLLMs reveal a substantial gap between answers and reasoning, especially on challenging tasks. Further analysis shows that task-adaptive dimension selection and weighting better align evaluation results with expert judgments. Overall, our benchmark and metric support a more expert-aligned assessment of CCH understanding and offer a transferable reference for broader evaluations of cultural heritage. We publicly release the data, code, and evaluation scripts at https://github.com/ZevTsao/CulMind to facilitate reproducible research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
