解读影响思维链效能的因素：概率、记忆与噪声推理
Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoning
AI总结：
本文通过移位密码解码任务研究影响思维链推理效能的因素，在GPT-4、Claude 3和Llama 3.1上识别出输出概率、预训练记忆和中间操作噪声三个关键因素，并证明它们显著影响模型准确率。
摘要：
思维链（CoT）提示已被证明能增强大型语言模型（LLMs）的多步推理能力。然而，关于LLMs在给定CoT提示时是表现出抽象泛化还是依赖浅层启发式的争论仍在持续。为了理解影响CoT推理的因素，我们对解码移位密码这一符号推理任务进行了详细的案例研究，该任务中字母在字母表中向前移动一定步数。我们分析了三个LLMs——GPT-4、Claude 3和Llama 3.1——使用CoT提示执行此任务时产生的结果模式。通过聚焦于一个相对简单的单一任务，我们能够识别出三个系统性影响CoT性能的因素：任务预期输出的概率（概率）、模型在预训练期间隐式学到的内容（记忆）以及推理中涉及的中间操作数量（噪声推理）。我们表明，这些因素可以在所有三个LLMs中极大地影响任务准确率；例如，在使用GPT-4测试时，改变输出出现概率会使准确率从26%变化到70%。总体而言，我们得出结论：CoT提示性能既反映了记忆，也反映了真实推理的概率性版本。代码和数据见https://github.com/aksh555/deciphering_cot
英文摘要：
Chain-of-Thought (CoT) prompting has been shown to enhance the multi-step reasoning capabilities of Large Language Models (LLMs). However, debates persist about whether LLMs exhibit abstract generalization or rely on shallow heuristics when given CoT prompts. To understand the factors influencing CoT reasoning we provide a detailed case study of the symbolic reasoning task of decoding shift ciphers, where letters are shifted forward some number of steps in the alphabet. We analyze the pattern of results produced by three LLMs -- GPT-4, Claude 3, and Llama 3.1 -- performing this task using CoT prompting. By focusing on a single relatively simple task, we are able to identify three factors that systematically affect CoT performance: the probability of the task's expected output (probability), what the model has implicitly learned during pre-training (memorization), and the number of intermediate operations involved in reasoning (noisy reasoning). We show that these factors can drastically influence task accuracy across all three LLMs; e.g., when tested with GPT-4, varying the output's probability of occurrence shifts accuracy from 26% to 70%. Overall, we conclude that CoT prompting performance reflects both memorization and a probabilistic version of genuine reasoning. Code and data at this https://github.com/aksh555/deciphering_cot
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
