基于变换器、RNN和思维链的组合推理
Compositional Reasoning with Transformers, RNNs, and Chain of Thought
AI总结：
本文研究了变换器、RNN和思维链架构在解决组合推理问题上的表现，证明了不同架构在处理此类问题时各有优劣，无一严格优于其他。
摘要：
已知不同的神经网络架构适用于不同的任务，但给定任务是否存在单一最佳架构？我们比较了变换器、RNN和带有思维链标记的变换器在一组我们称为组合推理问题（CRQ）的简单自然任务上的表达能力。这一类任务捕捉了具有树状组合结构的多步骤问题，例如评估布尔公式。我们证明，在标准难度假设下，除非某些超参数（深度、嵌入维度和思维链标记数量）随输入规模增长，否则这三种架构都无法解决CRQs。随后，我们为每种架构提供了解决CRQs的构造方法。对于变换器，我们的构造使用与问题规模对数成比例的深度。对于RNNs，需要且足够的是对数嵌入维度，只要输入以特定顺序提供。对于带有思维链的变换器，我们的构造使用n个思维链标记作为输入规模n。这些结果表明，尽管CRQs本质上困难，但语言模型有几种不同的方法来克服这种难度。即使对于单一类问题，每种架构都有其优势和劣势，没有一种严格优于其他。
英文摘要：
It is well understood that different neural network architectures are suited to different tasks, but is there always a single best architecture for a given task? We compare the expressive power of transformers, RNNs, and transformers with chain of thought tokens on a simple and natural class of tasks we term Compositional Reasoning Questions (CRQ). This family captures multi-step problems with tree-like compositional structure, such as evaluating Boolean formulas. We prove that under standard hardness assumptions, \emph{none} of these three architectures is capable of solving CRQs unless some hyperparameter (depth, embedding dimension, and number of chain of thought tokens, respectively) grows with the size of the input. We then provide constructions for solving CRQs with each architecture. For transformers, our construction uses depth that is logarithmic in the problem size. For RNNs, logarithmic embedding dimension is necessary and sufficient, so long as the inputs are provided in a certain order. For transformers with chain of thought, our construction uses $n$ CoT tokens for input size $n$. These results show that, while CRQs are inherently hard, there are several different ways for language models to overcome this hardness. Even for a single class of problems, each architecture has strengths and weaknesses, and none is strictly better than the others.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
