arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.13814cs.CLcs.AIcs.LG
收藏

ChatGPT在数学应用题上的独立评价(MWP)

An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP)

Arizona State University(亚利桑那州立大学)

Paulo Shakarian, Abhinav Koyyalamudi, Noel Ngu, Lakshmivihari Mareedu

摘要:

我们研究了一个商业上可用的大语言模型(LLM)被称为ChatGPT的数学单词问题(MWPs)从数据集DRAW-1 K的性能。据我们所知,这是ChatGPT的第一次独立评估。我们发现ChatGPT的性能根据显示其工作的需求而发生显着变化,当它提供工作时,有20%的时间失败,而当它不提供工作时,则有84%。与未知数的数量和操作的数量相关的关于MWP的其他几个因素导致与先前相比更高的失败概率,特别注意(在所有实验中)失败概率随着加法和减法操作的数量线性增加。我们还发布了ChatGPT对MWP的响应数据集,以支持LLM性能表征的进一步工作,并提供基线机器学习模型来预测ChatGPT是否可以正确回答MWP。我们发布了一个由ChatGPT的响应组成的数据集,以支持该领域的进一步研究。

英文摘要:

We study the performance of a commercially available large language model (LLM) known as ChatGPT on math word problems (MWPs) from the dataset DRAW-1K. To our knowledge, this is the first independent evaluation of ChatGPT. We found that ChatGPT's performance changes dramatically based on the requirement to show its work, failing 20% of the time when it provides work compared with 84% when it does not. Further several factors about MWPs relating to the number of unknowns and number of operations that lead to a higher probability of failure when compared with the prior, specifically noting (across all experiments) that the probability of failure increases linearly with the number of addition and subtraction operations. We also have released the dataset of ChatGPT's responses to the MWPs to support further work on the characterization of LLM performance and present baseline machine learning models to predict if ChatGPT can correctly answer an MWP. We have released a dataset comprised of ChatGPT's responses to support further research in this area.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑