自回归 + 思维链 = 循环：循环结构在语言模型可计算性中的作用及对循环 Transformer 的重新审视
Autoregressive + Chain of Thought = Recurrent: Recurrence's Role in Language Models' Computability and a Revisit of Recurrent Transformer
AI总结：
本研究探讨了循环结构对神经模型推理与可计算性的影响，指出思维链（CoT）通过模拟循环计算弥补了 Transformer 的局限，并提出“循环完备性”概念重新审视了 Linear Transformer 和 RWKV 等模型的理论缺陷。
摘要：
Transformer 架构在各种语言建模任务中表现优异，超越了 RNN 和 LSTM 等传统神经网络架构。这一定程度上得益于其消除了循环连接，从而允许并行训练并使梯度流动更平滑。然而，这种对循环结构的偏离使 Transformer 模型处于 Chomsky 计算层次的较低端，限制了其计算能力。因此，即使是先进的基于 Transformer 的模型在计数、字符串反转和乘法等任务上也面临相当大的困难。这些任务看似基础，但所需的计算复杂度超出了 Transformer 架构的能力。与此同时，“思维链”（CoT）提示的出现使基于 Transformer 的语言模型能够处理以前不可能或执行效果差的任务。在本研究中，我们深入探讨了神经模型中循环结构对其推理能力和可计算性的影响，并对比了自回归在神经模型计算能力中扮演的角色。随后，我们阐明了在语言模型背景下，CoT 方法如何模拟循环计算，并作为自回归与循环之间的桥梁。正是这种近似循环显著提升了模型的性能和计算能力。此外，我们重新审视了近期基于循环的 Transformer 模型设计，通过我们提出的“循环完备性”概念聚焦其计算能力，并指出了 Linear Transformer 和 RWKV 等模型的关键理论局限。通过这些工作，我们旨在为神经模型架构提供深刻见解，并促进更好的模型设计。
英文摘要：
The Transformer architecture excels in a variety of language modeling tasks, outperforming traditional neural architectures such as RNN and LSTM. This is partially due to its elimination of recurrent connections, which allows for parallel training and a smoother flow of gradients. However, this move away from recurrent structures places the Transformer model at the lower end of Chomsky's computational hierarchy, imposing limitations on its computational abilities. Consequently, even advanced Transformer-based models face considerable difficulties in tasks like counting, string reversal, and multiplication. These tasks, though seemingly elementary, require a level of computational complexity that exceeds the capabilities of the Transformer architecture. Concurrently, the emergence of ``Chain of Thought" (CoT) prompting has enabled Transformer-based language models to tackle tasks that were previously impossible or poorly executed. In this work, we thoroughly investigate the influence of recurrent structures in neural models on their reasoning abilities and computability, contrasting the role autoregression plays in the neural models' computational power. We then shed light on how the CoT approach can mimic recurrent computation and act as a bridge between autoregression and recurrence in the context of language models. It is this approximated recurrence that notably improves the model's performance and computational capacity. Moreover, we revisit recent recurrent-based Transformer model designs, focusing on their computational abilities through our proposed concept of ``recurrence-completeness" and identify key theoretical limitations in models like Linear Transformer and RWKV. Through this, we aim to provide insight into the neural model architectures and prompt better model design.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
