思维链的轻率？对规划中思维链的分析
Chain of Thoughtlessness? An Analysis of CoT in Planning
AI总结：
本文通过Blocksworld等规划任务分析思维链提示，发现其性能提升仅在提示与问题高度特定匹配时出现，且随问题规模增大迅速失效，表明思维链并未使模型学会通用算法，而是依赖精心设计的特定提示，并揭示了性能提升与人力成本之间的尖锐权衡。
摘要：
大型语言模型（LLM）在推理问题上的性能通常无法在分布外泛化。此前的工作声称，可以通过思维链提示（一种演示求解过程的方法）来缓解这一问题，其直觉是可以在上下文内教会LLM一种解决问题的算法。本文以经典规划领域Blocksworld中的问题为案例，研究思维链，并从两个维度考察两个最先进的LLM的性能：提示中给出的示例的通用性，以及每个提示所查询问题的复杂性。虽然我们的问题非常简单，但我们只在思维链提示与其问题类别极其特定地匹配时才发现有意义的性能提升，并且当查询指定的栈的大小n超过示例中所示栈的大小时，这些提升会迅速恶化。我们还创建了以往思维链论文中常研究的三个领域的可扩展变体，并证明了类似失败模式的存在。我们的结果暗示，与文献中先前的说法相反，思维链的性能提升并非源于模型通过演示学习通用算法程序，而是依赖于精心设计高度针对特定问题的提示。这凸显了思维链的缺点，尤其是可能的性能提升与生成具有正确推理轨迹的示例所需的人力劳动之间的尖锐权衡。
英文摘要：
Large language model (LLM) performance on reasoning problems typically does not generalize out of distribution. Previous work has claimed that this can be mitigated with chain of thought prompting-a method of demonstrating solution procedures-with the intuition that it is possible to in-context teach an LLM an algorithm for solving the problem. This paper presents a case study of chain of thought on problems from Blocksworld, a classical planning domain, and examines the performance of two state-of-the-art LLMs across two axes: generality of examples given in prompt, and complexity of problems queried with each prompt. While our problems are very simple, we only find meaningful performance improvements from chain of thought prompts when those prompts are exceedingly specific to their problem class, and that those improvements quickly deteriorate as the size n of the query-specified stack grows past the size of stacks shown in the examples. We also create scalable variants of three domains commonly studied in previous CoT papers and demonstrate the existence of similar failure modes. Our results hint that, contrary to previous claims in the literature, CoT's performance improvements do not stem from the model learning general algorithmic procedures via demonstrations but depend on carefully engineering highly problem specific prompts. This spotlights drawbacks of chain of thought, especially the sharp tradeoff between possible performance gains and the amount of human labor necessary to generate examples with correct reasoning traces.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
