arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.12744cs.CL
收藏

指导大型语言模型识别并忽略无关条件

Instructing Large Language Models to Identify and Ignore Irrelevant Conditions

Xi’an Jiaotong University(西安交通大学);University of Notre Dame(圣母大学)

Zhenyu Wu, Chao Shen, Meng Jiang

AI总结:

针对数学应用题中无关条件干扰LLM推理的问题,提出I³C方法,通过识别、验证并忽略无关条件,结合少样本选择,在多个数据集上显著提升准确率。

摘要:

数学应用题(MWP)求解需要根据给定的问题描述生成推理路径,而问题描述中往往包含无关条件。现有的思维链(CoT)提示方法激发了大型语言模型(LLM)的多步推理能力来解决MWP。然而,这些方法会被无关条件严重干扰,导致准确率较低。在本文中,我们提出了一种名为I$^3$C的新方法,指导LLM识别并忽略无关条件。该方法首先识别出一组与问题语义相关性较弱的无关条件候选,然后提示LLM验证这些无关条件,最后利用对相关和无关条件的验证结果来指导LLM,以避免混淆并改进推理路径。此外,我们提出选择(问题,推理路径)对作为示例,以增强I$^3$C的少样本推理能力。我们开发了I$^3$C-Select,基于语义相关性度量选择最易混淆的问题。我们在八个MWP数据集上进行了广泛实验。I$^3$C可以与任何CoT提示方法结合,以提高求解MWP的性能。值得注意的是,使用GPT-3.5-Turbo和I$^3$C-Select,我们在GSM-IC2-1K和GSM-ICM-1K上分别达到了96.0和94.1的准确率,显著超过了最先进的少样本提示方法Complex-CoT,分别提升了+11.7和+11.1。我们的实现已在https://wzy6642.github.io/I3C.github.io/公开提供。

英文摘要:

Math word problem (MWP) solving requires generating a reasoning path based on a given problem description that often contains irrelevant conditions. Existing chain-of-thought (CoT) prompting methods elicited multi-step reasoning abilities of large language models (LLMs) to solve MWPs. However, they were seriously confused by the irrelevant conditions, resulting in low accuracy. In this paper, we propose a novel approach named I$^3$C that instructs LLMs to identify and ignore irrelevant conditions. It identifies a set of irrelevant condition candidates that have a weak semantic relevance with the question. Then it prompts LLMs to verify the irrelevant conditions. Lastly it instructs the LLMs with the verification on relevant and irrelevant conditions to avoid confusion and improve reasoning paths. Moreover, we propose to select (problem, reasoning paths) pairs as demonstrations to enhance I$^3$C with few-shot reasoning. We develop I$^3$C-Select that selects the most confusing problems based on the semantic relevance measurement. We conduct extensive experiments on eight MWP datasets. I$^3$C can be combined with any CoT prompting methods to improve the performance of solving MWPs. Notably, with GPT-3.5-Turbo and I$^3$C-Select, we achieve an accuracy of 96.0 and 94.1 on GSM-IC2-1K and GSM-ICM-1K, respectively, significantly outperforming the state-of-the-art few-shot prompting method Complex-CoT by +11.7 and +11.1. Our implementation is made publicly available at https://wzy6642.github.io/I3C.github.io/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑