arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.02864cs.CLcs.AI
收藏

NUMCoT:使用大型语言模型进行思维链推理中的数字与度量单位

NUMCoT: Numerals and Units of Measurement in Chain-of-Thought Reasoning using Large Language Models

Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);School of Computing and Information Systems, Singapore Management University(新加坡管理大学计算与信息系统学院);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))

Ancheng Xu, Minghuan Tan, Lei Wang, Min Yang, Ruifeng Xu

AI总结:

本文通过构建带有扰动的数据集,评估大型语言模型在处理数字与度量单位转换方面的能力,发现微小变化会显著影响模型性能,且现有模型在此类转换上仍存在困难。

摘要:

数字系统和度量单位是人类活动中的两个紧密相连的主题,并与表达它们的语言相互影响。目前,对大型语言模型(LLMs)的评估通常涉及数学推理,但很少有人关注数字或单位的微小变化如何极大地改变问题的复杂性和LLMs的性能。在本文中,我们通过构建带有扰动的数据集来仔细审查现有LLMs在处理数字和度量单位方面的表现。我们首先将数学应用题的推理解剖为不同的子过程,如从语言到数字的数字转换和基于单位的度量转换。然后,我们进一步标注了来自中国古代算术著作的数学应用题,这些题目在数字和度量单位方面具有挑战性。在扰动数据集上的实验表明,LLMs在处理数字和度量转换方面仍然遇到困难。

英文摘要:

Numeral systems and units of measurement are two conjoined topics in activities of human beings and have mutual effects with the languages expressing them. Currently, the evaluation of Large Language Models (LLMs) often involves mathematical reasoning, yet little attention is given to how minor changes in numbers or units can drastically alter the complexity of problems and the performance of LLMs. In this paper, we scrutinize existing LLMs on processing of numerals and units of measurement by constructing datasets with perturbations. We first anatomize the reasoning of math word problems to different sub-procedures like numeral conversions from language to numbers and measurement conversions based on units. Then we further annotate math word problems from ancient Chinese arithmetic works which are challenging in numerals and units of measurement. Experiments on perturbed datasets demonstrate that LLMs still encounter difficulties in handling numeral and measurement conversions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑