arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.00344cs.CLcs.AIcs.IR
收藏

大型语言模型能精通数学吗?探究大型语言模型在 Math Stack Exchange 上的表现

Can LLMs Master Math? Investigating Large Language Models on Math Stack Exchange

FIZ Karlsruhe(弗劳恩霍夫 Karlsruhe 研究所);University of Göttingen(哥廷根大学);NII(信息学研究所(日本))

Ankit Satpute, Noah Giessing, Andre Greiner-Petter, Moritz Schubotz, Olaf Teschke, Akiko Aizawa, Bela Gipp

AI总结:

本文通过基准测试与人工案例分析评估LLMs在Math Stack Exchange数学问答上的表现,发现GPT-4虽优于现有方法但仍存在准确性与一致性的局限。

摘要:

大型语言模型(LLMs)在各类自然语言任务中展现了卓越的能力,其表现常常超越人类。尽管取得了这些进展,数学领域仍构成一项独特挑战,主要因其专门化的结构和对精确性的要求。在本研究中,我们采用两步方法来探究LLMs回答数学问题的能力。首先,我们使用在数学问答基准上表现最优的LLMs,为来自Math Stack Exchange(MSE)的78个问题生成答案。其次,对表现最佳的LLM进行案例分析,通过人工评估重点关注其答案的质量与准确性。我们发现,在现有针对数学问答微调的LLMs中,GPT-4表现最佳(nDCG为0.48,P@10为0.37),并且在考虑P@10的情况下,超越了ArqMATH3 Task1上当前最好的方法。我们的案例分析表明,尽管GPT-4在某些情况下能够生成相关的回答,但它并不能始终准确地回答所有问题。本文探讨了LLMs在处理复杂数学问题求解方面的当前局限。通过案例分析,我们揭示了LLMs在数学领域能力的不足,从而为未来AI驱动的数学推理研究与发展奠定基础。我们将代码和研究结果公开以供研究使用:\url{https://github.com/gipplab/LLM-Investig-MathStackExchange}

英文摘要:

Large Language Models (LLMs) have demonstrated exceptional capabilities in various natural language tasks, often achieving performances that surpass those of humans. Despite these advancements, the domain of mathematics presents a distinctive challenge, primarily due to its specialized structure and the precision it demands. In this study, we adopted a two-step approach for investigating the proficiency of LLMs in answering mathematical questions. First, we employ the most effective LLMs, as identified by their performance on math question-answer benchmarks, to generate answers to 78 questions from the Math Stack Exchange (MSE). Second, a case analysis is conducted on the LLM that showed the highest performance, focusing on the quality and accuracy of its answers through manual evaluation. We found that GPT-4 performs best (nDCG of 0.48 and P@10 of 0.37) amongst existing LLMs fine-tuned for answering mathematics questions and outperforms the current best approach on ArqMATH3 Task1, considering P@10. Our Case analysis indicates that while the GPT-4 can generate relevant responses in certain instances, it does not consistently answer all questions accurately. This paper explores the current limitations of LLMs in navigating complex mathematical problem-solving. Through case analysis, we shed light on the gaps in LLM capabilities within mathematics, thereby setting the stage for future research and advancements in AI-driven mathematical reasoning. We make our code and findings publicly available for research: \url{https://github.com/gipplab/LLM-Investig-MathStackExchange}

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑