EduArt：评估大型语言模型艺术史知识的教育级基准
EduArt: An educational-level benchmark for evaluating art history knowledge in large language models
AI总结：
提出EduArt基准，包含871道人工出题的艺术史题目，评估多模态大模型在不同格式和语言下的知识掌握与推理能力，发现格式显著影响表现，多选题高估模型能力。
摘要：
大型语言模型在通用基准测试中得分接近天花板，但这些聚合指标几乎无法揭示模型在单一学科内的行为。现有的艺术类评估依赖合成问题，很少报告题目层面的属性。本文介绍了EduArt，一个用于评估多模态大模型艺术史知识和视觉推理的教育级基准。EduArt包含来自意大利中学练习和美国大学先修艺术史考试的871道人工出题，涵盖两种语言和七种格式，从多选题到文本内单词放置和错误识别。对来自六个提供商家族的十二个模型进行了评估，包括默认仅答案条件和需要书面理由的动机条件，并使用经典测试理论和逻辑回归分析格式、语言、图像存在和模型的影响。该基准显示出强大的心理测量特性（平均区分度0.514，82.3%的良好区分度），而六个模型的多选题准确率接近饱和，表明仅识别格式无法区分前沿模型。格式是准确率的强独立预测因子：在多选题上超过94%的模型在开放式完成题上降至23.9%（Claude Opus 4.6）和错误识别题上降至6.2%（Claude Sonnet 4.6）。动机条件以主要负面且依赖模型家族的方式改变了准确率。这些分离表明艺术史知识和运用知识的能力是不同的能力，单格式基准高估了模型的可靠能力。绘制这一能力概况是负责任地在艺术史学术中使用多模态大模型的先决条件，因为该领域的任务要求生成和操作内容，而非从固定选项中选择。
英文摘要：
Large language models now score near ceiling on general benchmarks, but these aggregate measures reveal little about how models behave within single disciplines. Existing art-focused evaluations rely on synthetic questions and rarely report item-level properties. This paper introduces EduArt, an educational-level benchmark for art-historical knowledge and visual reasoning in multimodal LLMs. EduArt comprises 871 human-authored questions from Italian secondary-school exercises and US Advanced Placement Art History exams, spanning two languages and seven formats from multiple choice to in-text word placement and error identification. Twelve models from six provider families were evaluated under a default answer-only condition and a motivation condition requiring written justification, and characterized using Classical Test Theory and a logistic regression isolating the effects of format, language, image presence, and model. The benchmark showed strong psychometric properties (mean discrimination 0.514, 82.3 percent good discriminators), while multiple-choice accuracy saturated near ceiling for six models, showing recognition formats alone cannot distinguish frontier models. Format was a strong independent predictor of accuracy: models exceeding 94 percent on multiple choice fell to 23.9 percent on open completion (Claude Opus 4.6) and 6.2 percent on error identification (Claude Sonnet 4.6). The motivation condition changed accuracy in a predominantly negative, family-dependent direction. These dissociations indicate that art-historical knowledge and the ability to deploy it are distinct capabilities, and that single-format benchmarks overestimate what models can reliably do. Mapping this capability profile is a precondition for responsible use of multimodal LLMs in art-historical scholarship, where tasks demand producing and manipulating content rather than selecting from fixed options.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
