暴露阿喀琉斯之踵：评估大语言模型处理数学推理中错误的能力
Exposing the Achilles' Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoning
AI总结：
本文通过构建包含正确与错误推理步骤的MWP-MISTAKE数据集，系统评估了GPT-4o等大语言模型在数学推理中检测和纠正错误的能力，揭示了数据污染问题并提出了改进方向。
摘要：
大语言模型（LLMs）已被应用于数学应用题（MWPs），产生了变革性影响，彻底改变了这些复杂问题在包括教育环境在内的各个领域中的处理与求解方式。然而，对这些模型的评估往往优先考虑最终准确率，忽视了推理能力这一关键方面。本研究通过聚焦于LLMs检测和纠正推理错误的能力来填补这一空白。我们引入了一个新数据集MWP-MISTAKE，其中包含通过基于规则的方法和较小语言模型生成的带有正确和错误推理步骤的数学应用题。我们的全面基准评测揭示了最先进模型（如GPT-4o、GPT-4、GPT-3.5Turbo等）的优势与不足。我们强调了GPT-4o在错误检测和纠正方面的卓越表现，以及较小模型所面临的持续挑战。此外，我们识别了与数据污染和记忆相关的问题，这些问题影响了LLMs在实际应用中的可靠性。我们的发现强调了严格评估推理过程的重要性，并提出了未来方向，以增强LLMs在数学问题求解中的泛化能力和鲁棒性。
英文摘要：
Large Language Models (LLMs) have been applied to Math Word Problems (MWPs) with transformative impacts, revolutionizing how these complex problems are approached and solved in various domains including educational settings. However, the evaluation of these models often prioritizes final accuracy, overlooking the crucial aspect of reasoning capabilities. This work addresses this gap by focusing on the ability of LLMs to detect and correct reasoning mistakes. We introduce a novel dataset MWP-MISTAKE, incorporating MWPs with both correct and incorrect reasoning steps generated through rule-based methods and smaller language models. Our comprehensive benchmarking reveals significant insights into the strengths and weaknesses of state-of-the-art models, such as GPT-4o, GPT-4, GPT-3.5Turbo, and others. We highlight GPT-$o's superior performance in mistake detection and rectification and the persistent challenges faced by smaller models. Additionally, we identify issues related to data contamination and memorization, impacting the reliability of LLMs in real-world applications. Our findings emphasize the importance of rigorous evaluation of reasoning processes and propose future directions to enhance the generalization and robustness of LLMs in mathematical problem-solving.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
