基于迭代式大语言模型生成与精炼数学应用题中的干扰条件
Iterative LLM-Based Generation and Refinement of Distracting Conditions in Math Word Problems
AI总结:
针对现有带干扰条件数学应用题数据集稀缺且易识别的问题,提出一个迭代式LLM框架,从多角度生成不改变原解答的干扰条件,高效且保持数据质量。
摘要:
数学推理是检验大语言模型(LLMs)智能水平的关键测试平台,而数学应用题(MWPs)是一类常见的数学问题。大多数MWP数据集仅包含必要信息的问题,而带有干扰性和多余条件的问题往往被忽视。先前的研究测试了流行的LLMs,发现在存在干扰条件时性能会急剧下降。然而,带有干扰条件的MWP数据集数量有限,且多数存在难度较低和表达脱离上下文的问题。这使得干扰条件容易被识别和排除,从而降低了基于这些数据集进行基准测试的可信度。此外,在添加干扰条件时,推理过程和答案也可能发生变化,需要大量人工来检查和编写解答。为解决这些问题,我们设计了一个使用LLMs生成干扰条件的迭代框架。我们开发了一组提示词,从不同角度和认知水平修改MWPs,鼓励生成干扰条件以及进一步修改的建议。另一个优势是原始问题和修改后问题共享解答:我们明确引导LLMs生成不改变原始解答的干扰条件,从而避免生成新解答的需要。该框架高效且易于部署,在保持数据质量的同时,降低了生成带干扰条件MWPs的开销。
英文摘要:
Mathematical reasoning serves as a crucial testbed for the intelligence of large language models (LLMs), and math word problems (MWPs) are a popular type of math problems. Most MWP datasets consist of problems containing only the necessary information, while problems with distracting and excessive conditions are often overlooked. Prior works have tested popular LLMs and found a dramatic performance drop in the presence of distracting conditions. However, datasets of MWPs with distracting conditions are limited, and most suffer from lower levels of difficulty and out-of-context expressions. This makes distracting conditions easy to identify and exclude, thus reducing the credibility of benchmarking on them. Moreover, when adding distracting conditions, the reasoning and answers may also change, requiring intensive labor to check and write the solutions. To address these issues, we design an iterative framework to generate distracting conditions using LLMs. We develop a set of prompts to revise MWPs from different perspectives and cognitive levels, encouraging the generation of distracting conditions as well as suggestions for further revision. Another advantage is the shared solutions between original and revised problems: we explicitly guide the LLMs to generate distracting conditions that do not alter the original solutions, thus avoiding the need to generate new solutions. This framework is efficient and easy to deploy, reducing the overhead of generating MWPs with distracting conditions while maintaining data quality.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。