剖析思维链：通过上下文过滤与学习实现组合性
Dissecting Chain-of-Thought: Compositionality through In-Context Filtering and Learning
AI总结：
本研究以多层感知机为对象，通过实验与理论分析揭示思维链通过分阶段数据过滤与单步学习降低上下文学习样本复杂度的机制，还发现其能加速预训练，为理解思维链推理原理提供新洞见。
摘要：
思维链（Chain-of-Thought, CoT）是一种让语言模型将复杂推理任务拆解为更简单步骤以完成处理的方法。尽管它已取得成功，但CoT的内在机制尚未被完全理解。为阐明这一问题，本研究探究了CoT对Transformer在上下文中学习一类简单易研究但具备通用性的组合函数——多层感知机（multi-layer perceptrons, MLPs）——的能力的影响。\n在该研究设置下，我们发现CoT的成功可归因于它将组合函数的上下文学习拆解为两个不同阶段：聚焦并过滤与组合的每个步骤相关的数据，以及在上下文中学习单步组合函数。通过实验与理论双重证据，我们证明了CoT如何显著降低上下文学习（in-context learning, ICL）的样本复杂度，并助力学习非CoT方法难以掌握的复杂函数。此外，我们阐明了Transformer只需加入额外层、通过注意力机制执行CoT所需的数据过滤，就能从普通上下文学习过渡到借助CoT掌握组合函数。\n除了这些测试阶段的优势，我们还表明CoT能通过学习表征复杂函数的捷径加速预训练，且过滤在这一过程中发挥着重要作用。这些发现共同为理解CoT的机制提供了洞见，也推动了对其在复杂推理任务中作用的进一步研究。
英文摘要：
Chain-of-thought (CoT) is a method that enables language models to handle complex reasoning tasks by decomposing them into simpler steps. Despite its success, the underlying mechanics of CoT are not yet fully understood. In an attempt to shed light on this, our study investigates the impact of CoT on the ability of transformers to in-context learn a simple to study, yet general family of compositional functions: multi-layer perceptrons (MLPs). In this setting, we find that the success of CoT can be attributed to breaking down in-context learning of a compositional function into two distinct phases: focusing on and filtering data related to each step of the composition and in-context learning the single-step composition function. Through both experimental and theoretical evidence, we demonstrate how CoT significantly reduces the sample complexity of in-context learning (ICL) and facilitates the learning of complex functions that non-CoT methods struggle with. Furthermore, we illustrate how transformers can transition from vanilla in-context learning to mastering a compositional function with CoT by simply incorporating additional layers that perform the necessary data-filtering for CoT via the attention mechanism. In addition to these test-time benefits, we show CoT helps accelerate pretraining by learning shortcuts to represent complex functions and filtering plays an important role in this process. These findings collectively provide insights into the mechanics of CoT, inviting further investigation of its role in complex reasoning tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
