arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.14804cs.CL
收藏

LLMs能否更好地解决更长的数学应用题?

Can LLMs Solve longer Math Word Problems Better?

The Hong Kong University of Science and Technology(香港科技大学);University of Science and Technology of China(中国科学技术大学)

Xin Xu, Tong Xiao, Zitong Chao, Zhenya Huang, Can Yang, Yang Wang

AI总结:

本研究提出上下文长度泛化性(CoLeG)概念及E-GSM数据集,针对LLMs解决长数学应用题能力不足,分别对专有和开源模型采用新提示与辅助微调任务,有效提升推理效能。

摘要:

数学应用题(MWPs)在评估大型语言模型(LLMs)的能力中扮演着至关重要的角色,然而当前的研究主要集中于上下文简洁的问题。较长上下文对数学推理的影响仍未得到充分探索。本研究率先探讨了上下文长度泛化性(CoLeG),即LLMs解决具有扩展叙述的MWPs的能力。我们引入了扩展小学数学(E-GSM),这是一个包含冗长叙述的MWPs集合,并提出了两个新颖的指标来评估LLMs在解决这些问题时的有效性和韧性。我们对现有的零样本提示技术与专有LLMs以及开源LLMs的分析揭示了CoLeG方面的普遍不足。为了缓解这些问题,我们针对不同类别的LLMs提出了定制方法。对于专有LLMs,我们引入了一种新的指令提示,旨在减轻长上下文的影响。对于开源LLMs,我们开发了一种新颖的辅助任务用于微调以增强CoLeG。我们的综合结果证明了所提方法的有效性,显示了在E-GSM上的性能提升。此外,我们进行了深入分析以区分语义理解和推理效能的影响,表明我们的方法改善了后者。我们还建立了我们的方法在几个其他MWP基准上的泛化性。我们的发现突显了当前LLMs的局限性,并相应提供了实用的解决方案,为模型泛化性和训练方法的进一步探索铺平了道路。

英文摘要:

Math Word Problems (MWPs) play a vital role in assessing the capabilities of Large Language Models (LLMs), yet current research primarily focuses on questions with concise contexts. The impact of longer contexts on mathematical reasoning remains under-explored. This study pioneers the investigation of Context Length Generalizability (CoLeG), which refers to the ability of LLMs to solve MWPs with extended narratives. We introduce Extended Grade-School Math (E-GSM), a collection of MWPs featuring lengthy narratives, and propose two novel metrics to evaluate the efficacy and resilience of LLMs in tackling these problems. Our analysis of existing zero-shot prompting techniques with proprietary LLMs along with open-source LLMs reveals a general deficiency in CoLeG. To alleviate these issues, we propose tailored approaches for different categories of LLMs. For proprietary LLMs, we introduce a new instructional prompt designed to mitigate the impact of long contexts. For open-source LLMs, we develop a novel auxiliary task for fine-tuning to enhance CoLeG. Our comprehensive results demonstrate the effectiveness of our proposed methods, showing improved performance on E-GSM. Additionally, we conduct an in-depth analysis to differentiate the effects of semantic understanding and reasoning efficacy, showing that our methods improves the latter. We also establish the generalizability of our methods across several other MWP benchmarks. Our findings highlight the limitations of current LLMs and offer practical solutions correspondingly, paving the way for further exploration of model generalizability and training methodologies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑