视觉语言模型能否评估手写数学？
Can Vision-Language Models Evaluate Handwritten Math?
AI总结：
本文提出FERMAT基准，覆盖2200余份手写数学解答，系统评估视觉语言模型在错误检测、定位和纠正上的能力，揭示现有模型在手写内容推理方面的不足并开源全部资源。
摘要：
视觉语言模型（VLMs）的最新进展为自动评分手写学生作答开辟了新的可能性，尤其是在数学领域。然而，目前仍缺乏一项全面研究来检验视觉语言模型评估和推理手写内容的能力。为填补这一空白，我们提出了FERMAT，一个旨在评估视觉语言模型检测、定位和纠正手写数学内容中错误能力的基准。FERMAT涵盖四个关键错误维度——计算性、概念性、符号性和呈现性——并包含超过2200份手写数学解答，这些解答源自609道经人工整理的7至12年级问题，并有意引入了扰动。利用FERMAT，我们在三个任务上对九个视觉语言模型进行了基准测试：错误检测、错误定位和错误纠正。我们的结果揭示了当前视觉语言模型在推理手写文本方面存在显著不足，其中Gemini-1.5-Pro取得了最高的错误纠正率（77%）。我们还观察到，一些模型在处理手写内容时存在困难，因为当手写输入被替换为印刷文本或图像时，它们的准确率有所提高。这些发现凸显了当前视觉语言模型的局限性，并揭示了新的改进方向。我们以开源形式发布FERMAT及所有相关资源，以推动进一步研究。
英文摘要：
Recent advancements in Vision-Language Models (VLMs) have opened new possibilities in automatic grading of handwritten student responses, particularly in mathematics. However, a comprehensive study to test the ability of VLMs to evaluate and reason over handwritten content remains absent. To address this gap, we introduce FERMAT, a benchmark designed to assess the ability of VLMs to detect, localize and correct errors in handwritten mathematical content. FERMAT spans four key error dimensions - computational, conceptual, notational, and presentation - and comprises over 2,200 handwritten math solutions derived from 609 manually curated problems from grades 7-12 with intentionally introduced perturbations. Using FERMAT we benchmark nine VLMs across three tasks: error detection, localization, and correction. Our results reveal significant shortcomings in current VLMs in reasoning over handwritten text, with Gemini-1.5-Pro achieving the highest error correction rate (77%). We also observed that some models struggle with processing handwritten content, as their accuracy improves when handwritten inputs are replaced with printed text or images. These findings highlight the limitations of current VLMs and reveal new avenues for improvement. We release FERMAT and all the associated resources in the open-source to drive further research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
