Critical-Questions-of-Thought：通过论证式查询引导LLM推理
Critical-Questions-of-Thought: Steering LLM reasoning with Argumentative Querying
AI总结：
本文针对LLM逻辑与数学推理泛化能力弱的问题，提出基于Toulmin论证模型中“批判性问题”的推理引导方法，通过探究推理过程纠正逻辑错误，在MT-Bench上表现优于基线及CoT。
摘要：
研究强调了尽管AI研究近期取得了突破和快速进展，但即使是最先进的大型语言模型（LLMs）在执行逻辑和数学推理时仍然面临困难。结果似乎表明，LLMs仍然作为（高度先进的）数据模式识别器运作，在尝试泛化并解决模型从未见过或与其训练数据中样本不接近的推理问题时得分很低。为了解决这一紧迫问题，本文利用了论证理论文献中“批判性问题”的概念，特别关注Toulmin论证模型。我们展示了运用这些批判性问题能够提升LLMs的推理能力。通过探究模型推理过程背后的基本原理，LLM能够评估是否出现了逻辑错误，并在向用户提示提供最终回复之前予以纠正。其基本思想源自任何有效论证程序的黄金标准：结论有效的前提是它被可接受的前提所蕴含。或者，用现实世界近似（以不完整信息和推定逻辑为特征）来转述这一亚里士多德原则，结论有效的前提是未被证明为无效。这种方法通过推理管道成功引导模型输出，相较于基线及其Chain-of-Thought (CoT)实现取得了更好的性能。为此，本文在一系列LLMs上对所提方法在MT-Bench Reasoning和Math任务上进行了广泛评估。
英文摘要：
Studies have underscored how, regardless of the recent breakthrough and swift advances in AI research, even state-of-the-art Large Language models (LLMs) continue to struggle when performing logical and mathematical reasoning. The results seem to suggest that LLMs still work as (highly advanced) data pattern identifiers, scoring poorly when attempting to generalise and solve reasoning problems the models have never previously seen or that are not close to samples presented in their training data. To address this compelling concern, this paper makes use of the notion of critical questions from the literature on argumentation theory, focusing in particular on Toulmin's model of argumentation. We show that employing these critical questions can improve the reasoning capabilities of LLMs. By probing the rationale behind the models' reasoning process, the LLM can assess whether some logical mistake is occurring and correct it before providing the final reply to the user prompt. The underlying idea is drawn from the gold standard of any valid argumentative procedure: the conclusion is valid if it is entailed by accepted premises. Or, to paraphrase such Aristotelian principle in a real-world approximation, characterised by incomplete information and presumptive logic, the conclusion is valid if not proved otherwise. This approach successfully steers the models' output through a reasoning pipeline, resulting in better performance against the baseline and its Chain-of-Thought (CoT) implementation. To this end, an extensive evaluation of the proposed approach on the MT-Bench Reasoning and Math tasks across a range of LLMs is provided.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
