SketchJudge: 一种用于用多模态大语言模型评分手绘图的诊断基准
SketchJudge: A Diagnostic Benchmark for Grading Hand-drawn Diagrams with Multimodal Large Language Models
AI总结：
SketchJudge是一个用于评估多模态大语言模型在评分手绘图任务中诊断能力的基准，通过1015份手绘学生回答验证了当前视觉-语言对齐在符号和嘈杂环境中的脆弱性。
摘要：
尽管多模态大语言模型（MLLMs）在视觉理解方面取得了显著进展，但它们在面对人类生成的无结构和模糊的草图时往往表现不佳。这种局限性在未被充分探索的视觉评分任务中尤为明显，其中模型不仅要解决一个问题，还要诊断手绘图中的错误。这种诊断能力依赖于复杂的结构性、语义性和元认知推理。为弥合这一差距，我们引入了SketchJudge，一种专门用于评估MLLMs作为手绘STEM图评分者的基准。SketchJudge涵盖了来自四个领域的1,015份手绘学生回答：几何学、物理学、图表和流程图，具有多样的风格变化和不同的错误类型。在SketchJudge上的评估表明，即使先进的MLLMs在很大程度上仍落后于人类，验证了该基准在暴露当前视觉-语言对齐在符号和嘈杂环境中的脆弱性方面的有效性。所有数据、代码和评估脚本均在https://github.com/yuhangsu82/SketchJudge上公开。
英文摘要：
While Multimodal Large Language Models (MLLMs) have achieved remarkable progress in visual understanding, they often struggle when faced with the unstructured and ambiguous nature of human-generated sketches. This limitation is particularly pronounced in the underexplored task of visual grading, where models should not only solve a problem but also diagnose errors in hand-drawn diagrams. Such diagnostic capabilities depend on complex structural, semantic, and metacognitive reasoning. To bridge this gap, we introduce SketchJudge, a novel benchmark tailored for evaluating MLLMs as graders of hand-drawn STEM diagrams. SketchJudge encompasses 1,015 hand-drawn student responses across four domains: geometry, physics, charts, and flowcharts, featuring diverse stylistic variations and distinct error types. Evaluations on SketchJudge demonstrate that even advanced MLLMs lag significantly behind humans, validating the benchmark's effectiveness in exposing the fragility of current vision-language alignment in symbolic and noisy contexts. All data, code, and evaluation scripts are publicly available at https://github.com/yuhangsu82/SketchJudge.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
