思维链赋能 Transformer 解决内在串行问题
Chain of Thought Empowers Transformers to Solve Inherently Serial Problems
AI总结:
本文从理论上证明思维链(CoT)能赋予低深度Transformer执行内在串行计算的能力,使其突破常数深度并行计算的表达能力上界,并在实验中显著提升了置换群复合等串行任务的准确率。
摘要:
指导模型生成一系列中间步骤(即思维链,chain of thought, CoT)是提高大型语言模型(LLMs)在算术和符号推理任务上准确率的高效方法。然而,CoT背后的机制尚不清楚。本研究从表达能力的角度为仅解码器Transformer的CoT能力提供了理论理解。概念上,CoT赋予了模型执行内在串行计算的能力,而这正是Transformer所缺乏的,尤其是在深度较低时。给定输入长度$n$,先前的工作表明,具有有限精度$\mathsf{poly}(n)$嵌入大小的常数深度Transformer在没有CoT时只能解决$\mathsf{TC}^0$中的问题。我们首先为具有常数位精度的常数深度Transformer展示了更紧的表达能力上界,它们只能解决$\mathsf{AC}^0$($\mathsf{TC}^0$的真子集)中的问题。然而,通过$T$步CoT,使用常数位精度和$O(\log n)$嵌入大小的常数深度Transformer可以解决任何可由大小为$T$的布尔电路解决的问题。实验表明,启用CoT能显著提高对并行计算困难任务的准确率,包括置换群的复合、迭代平方和电路值问题,特别是对于低深度Transformer。
英文摘要:
Instructing the model to generate a sequence of intermediate steps, a.k.a., a chain of thought (CoT), is a highly effective method to improve the accuracy of large language models (LLMs) on arithmetics and symbolic reasoning tasks. However, the mechanism behind CoT remains unclear. This work provides a theoretical understanding of the power of CoT for decoder-only transformers through the lens of expressiveness. Conceptually, CoT empowers the model with the ability to perform inherently serial computation, which is otherwise lacking in transformers, especially when depth is low. Given input length $n$, previous works have shown that constant-depth transformers with finite precision $\mathsf{poly}(n)$ embedding size can only solve problems in $\mathsf{TC}^0$ without CoT. We first show an even tighter expressiveness upper bound for constant-depth transformers with constant-bit precision, which can only solve problems in $\mathsf{AC}^0$, a proper subset of $ \mathsf{TC}^0$. However, with $T$ steps of CoT, constant-depth transformers using constant-bit precision and $O(\log n)$ embedding size can solve any problem solvable by boolean circuits of size $T$. Empirically, enabling CoT dramatically improves the accuracy for tasks that are hard for parallel computation, including the composition of permutation groups, iterated squaring, and circuit value problems, especially for low-depth transformers.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。