arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.10839cs.CLcs.LG
收藏

面向数学推理任务的大型语言模型基准测试

Benchmarking Large Language Models for Math Reasoning Tasks

Technical University of Munich(慕尼黑工业大学)

Kathrin Seßler, Yao Rong, Emek Gözlüklü, Enkelejda Kasneci

AI总结:

本文提出了一个数学推理基准,在4个基础模型上跨5个数据集公平比较了7种上下文学习算法,发现大模型性能独立于提示策略而小模型受其显著影响,并开源了代码。

摘要:

在数学推理中使用大型语言模型(LLMs)已成为相关研究的基石,这展示了这些模型的智能,并通过其先进性能(例如在教育场景中)实现了潜在的实际应用。尽管设计了各种数据集和上下文学习算法来提高 LLMs 自动化数学问题求解的能力,但缺乏跨不同数据集的综合基准测试,使得为特定任务选择合适的模型变得复杂。在本项目中,我们提出了一个基准,在四个强大的基础模型上,跨五个广泛使用的数学数据集,公平地比较了七种最先进的数学问题求解上下文学习算法。此外,我们探讨了效率与性能之间的权衡,突出了 LLMs 在数学推理中的实际应用。我们的结果表明,像 GPT-4o 和 LLaMA 3-70B 这样的大型基础模型能够独立于具体的提示策略解决数学推理,而对于较小的模型,上下文学习方法会显著影响性能。此外,最佳提示取决于所选的基础模型。我们开源了基准代码,以支持未来研究中集成更多模型。

英文摘要:

The use of Large Language Models (LLMs) in mathematical reasoning has become a cornerstone of related research, demonstrating the intelligence of these models and enabling potential practical applications through their advanced performance, such as in educational settings. Despite the variety of datasets and in-context learning algorithms designed to improve the ability of LLMs to automate mathematical problem solving, the lack of comprehensive benchmarking across different datasets makes it complicated to select an appropriate model for specific tasks. In this project, we present a benchmark that fairly compares seven state-of-the-art in-context learning algorithms for mathematical problem solving across five widely used mathematical datasets on four powerful foundation models. Furthermore, we explore the trade-off between efficiency and performance, highlighting the practical applications of LLMs for mathematical reasoning. Our results indicate that larger foundation models like GPT-4o and LLaMA 3-70B can solve mathematical reasoning independently from the concrete prompting strategy, while for smaller models the in-context learning approach significantly influences the performance. Moreover, the optimal prompt depends on the chosen foundation model. We open-source our benchmark code to support the integration of additional models in future research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑