新手学习者与专家导师：基于错误概念评估大语言模型的数学推理能力
Novice Learner and Expert Tutor: Evaluating Math Reasoning Abilities of Large Language Models with Misconceptions
AI总结：
本研究提出基于数学错误概念的LLMs数学推理评估方法，模拟LLMs分别扮演新手学习者与专家导师，实验发现LLMs虽能正确答题却难识别错误答案及对应错误概念，为教育应用中提升LLMs提供新方向。
摘要：
我们提出了基于数学错误概念的大语言模型（LLMs）数学推理能力的新型评估方法。我们的核心方法是将LLMs分别模拟为新手学习者和专家导师，前者旨在识别由特定错误概念导致的数学题错误答案，后者旨在识别错误答案背后的错误概念。与传统的基于LLMs的数学评估聚焦于正确解答数学题不同，我们的方法借鉴了教育学习科学的原理。我们明确要求LLMs模仿新手学习者，基于不完整知识以特定的错误方式回答问题；同时模仿专家导师，识别与题目错误答案对应的错误概念。利用简单的小学数学题，我们的实验表明，尽管LLMs能够轻松正确解答这些问题，但它们难以做到两点：1）识别与特定不完整知识（错误概念）对应的错误答案；2）解释特定错误答案的错误概念。我们的研究为提升LLMs的数学推理能力指出了新方向，尤其是在智能辅导系统等教育应用中开发鲁棒的学生模拟和专家辅导模型方面。
英文摘要：
We propose novel evaluations for mathematical reasoning capabilities of Large Language Models (LLMs) based on mathematical misconceptions. Our primary approach is to simulate LLMs as a novice learner and an expert tutor, aiming to identify the incorrect answer to math question resulted from a specific misconception and to recognize the misconception(s) behind an incorrect answer, respectively. Contrary to traditional LLMs-based mathematical evaluations that focus on answering math questions correctly, our approach takes inspirations from principles in educational learning sciences. We explicitly ask LLMs to mimic a novice learner by answering questions in a specific incorrect manner based on incomplete knowledge; and to mimic an expert tutor by identifying misconception(s) corresponding to an incorrect answer to a question. Using simple grade-school math problems, our experiments reveal that, while LLMs can easily answer these questions correctly, they struggle to identify 1) the incorrect answer corresponding to specific incomplete knowledge (misconceptions); 2) the misconceptions that explain particular incorrect answers. Our study indicates new opportunities for enhancing LLMs' math reasoning capabilities, especially on developing robust student simulation and expert tutoring models in the educational applications such as intelligent tutoring systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
