Resprompt：残差连接提示推进大型语言模型中的多步推理
Resprompt: Residual Connection Prompting Advances Multi-Step Reasoning in Large Language Models
AI总结：
针对标准CoT难以表示多步推理图的问题，本文提出RESPROMPT，通过在提示中加入残差连接重构推理依赖，并在六个基准上显著提升LLaMA模型的多步推理准确率。
摘要：
思维链（CoT）提示通过提供逐步的问题解决理据，令人瞩目地释放了大型语言模型（LLM）的推理潜力。然而，标准CoT在需要多个推理步骤的问题上效果较差。这一局限源于多步问题中的复杂推理过程：后续阶段往往依赖于若干更早步骤的结果，而不仅仅是紧邻前一步的结果。这种复杂性表明推理过程天然可表示为图。然而，CoT提示近乎线性且直接的结构难以捕捉这种复杂推理图。为应对这一挑战，我们提出残差连接提示（RESPROMPT），这是一种推进LLM多步推理的新型提示策略。我们的核心思想是在提示中重构推理图。为此，我们将必要连接——即推理图中存在但在线性CoT流程中缺失的链接——整合进提示。这些链接被称为“残差连接”，对于将线性CoT结构转化为图表示至关重要，可有效捕捉多步问题内在的复杂推理图。我们在数学、序列和常识推理三个不同领域的六个基准上评估RESPROMPT。对于开源的LLaMA模型系列，RESPROMPT在LLaMA-65B上带来12.5%的显著平均推理准确率提升，在LLaMA2-70B上带来6.8%的提升。细分分析进一步表明，RESPROMPT尤其擅长复杂多步推理：对于至少需要五个推理步骤的问题，RESPROMPT相比最佳的基于CoT的基准，在LLaMA-65B上取得21.1%、在LLaMA2-70B上取得14.3%的可观平均提升。通过大量消融研究和分析，我们明确了如何最有效地构建残差连接。
英文摘要：
Chain-of-thought (CoT) prompting, which offers step-by-step problem-solving rationales, has impressively unlocked the reasoning potential of large language models (LLMs). Yet, the standard CoT is less effective in problems demanding multiple reasoning steps. This limitation arises from the complex reasoning process in multi-step problems: later stages often depend on the results of several steps earlier, not just the results of the immediately preceding step. Such complexities suggest the reasoning process is naturally represented as a graph. The almost linear and straightforward structure of CoT prompting, however, struggles to capture this complex reasoning graph. To address this challenge, we propose Residual Connection Prompting (RESPROMPT), a new prompting strategy that advances multi-step reasoning in LLMs. Our key idea is to reconstruct the reasoning graph within prompts. We achieve this by integrating necessary connections-links present in the reasoning graph but missing in the linear CoT flow-into the prompts. Termed "residual connections", these links are pivotal in morphing the linear CoT structure into a graph representation, effectively capturing the complex reasoning graphs inherent in multi-step problems. We evaluate RESPROMPT on six benchmarks across three diverse domains: math, sequential, and commonsense reasoning. For the open-sourced LLaMA family of models, RESPROMPT yields a significant average reasoning accuracy improvement of 12.5% on LLaMA-65B and 6.8% on LLaMA2-70B. Breakdown analysis further highlights RESPROMPT particularly excels in complex multi-step reasoning: for questions demanding at least five reasoning steps, RESPROMPT outperforms the best CoT based benchmarks by a remarkable average improvement of 21.1% on LLaMA-65B and 14.3% on LLaMA2-70B. Through extensive ablation studies and analyses, we pinpoint how to most effectively build residual connections.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
