超越最终答案：评估大语言模型的数学辅导能力
Beyond Final Answers: Evaluating Large Language Models for Math Tutoring
AI总结：
本研究提出两种评估大语言模型数学辅导能力的新方法，测试多款ChatGPT模型后发现其解题正确率较高但作为互动辅导者时完全正确率不足，指出其暂不能在无监督下用作数学智能导师。
摘要：
研究人员在应用大语言模型（LLM）解决数学问题方面取得了显著进展，这一点通过GSM8k、ProofNet、AlphaGeometry和MathOdyssey等项目的成果得到了印证。这一进展引发了人们对其在数学辅导学生方面应用潜力的兴趣。然而，在正确性和教学质量至关重要的辅导场景中，大语言模型的可靠性仍未得到充分探究。此外，大语言模型的解题能力未必能转化为对学生有效的辅导支持。在本研究中，我们提出了两种新颖的方法，用于评估大语言模型在数学辅导场景中的正确性和质量。第一种方法以一个面向大学代数的智能辅导系统为测试平台，评估大语言模型的解题能力。我们利用该辅导系统生成基准问题，提示多种不同的大语言模型求解这些问题，并将其解答与辅导系统生成的解答进行对比。第二种方法将大语言模型作为辅导者而非解题者进行评估。我们聘请人类评估人员扮演向各款大语言模型寻求辅导支持的学生，随后通过定性编码流程评估大语言模型所提供支持的质量与正确性。我们应用这些方法评估了多款ChatGPT模型，包括3.5 Turbo、4、4o、o1-mini和o1-preview。研究结果显示，当作为解题者使用时，大语言模型在测试的大学代数问题中，有85.5%能生成正确的最终答案。当以交互方式作为辅导者使用时，90%的大语言模型对话展现出高质量的教学支持；但其中许多对话存在错误——仅有56.6%的对话是完全正确的。我们得出结论：尽管大语言模型具备潜力，但在没有人类监督或额外的正确性与质量保障机制的情况下，它们尚不适合作为数学智能辅导工具。
英文摘要：
Researchers have made notable progress in applying Large Language Models (LLMs) to solve math problems, as demonstrated through efforts like GSM8k, ProofNet, AlphaGeometry, and MathOdyssey. This progress has sparked interest in their potential use for tutoring students in mathematics. However, the reliability of LLMs in tutoring contexts -- where correctness and instructional quality are crucial -- remains underexplored. Moreover, LLM problem-solving capabilities may not necessarily translate into effective tutoring support for students. In this work, we present two novel approaches to evaluate the correctness and quality of LLMs in math tutoring contexts. The first approach uses an intelligent tutoring system for college algebra as a testbed to assess LLM problem-solving capabilities. We generate benchmark problems using the tutor, prompt a diverse set of LLMs to solve them, and compare the solutions to those generated by the tutor. The second approach evaluates LLM as tutors rather than problem solvers. We employ human evaluators, who act as students seeking tutoring support from each LLM. We then assess the quality and correctness of the support provided by the LLMs via a qualitative coding process. We applied these methods to evaluate several ChatGPT models, including 3.5 Turbo, 4, 4o, o1-mini, and o1-preview. Our findings show that when used as problem solvers, LLMs generate correct final answers for 85.5% of the college algebra problems tested. When employed interactively as tutors, 90% of LLM dialogues show high-quality instructional support; however, many contain errors -- only 56.6% are entirely correct. We conclude that, despite their potential, LLMs are not yet suitable as intelligent tutors for math without human oversight or additional mechanisms to ensure correctness and quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
