低精度softmax变换器的表达能力（摘要）链式思维
The Expressive Power of Low Precision Softmax Transformers with (Summarized) Chain-of-Thought
AI总结：
本文研究了低精度softmax变换器在链式思维中的表达能力，通过构造三元激活和分离注意力分数的硬max变换器来模拟图灵机，从而将构造转换为等效的softmax变换器，并分析了最近提出的总结链式思维范式在模拟图灵机时的效率。
摘要：
现有的变换器表达性结果通常依赖于hardmax注意力、高精度和其它架构修改，这些修改将它们与实际使用的模型脱节。我们通过分析具有softmax注意力和激活值及注意力权重四舍五入的标准变换器解码器，同时允许深度和宽度以对数方式增长于上下文长度，来弥合这一差距。作为中间步骤，我们构造了具有三元激活和良好分离注意力分数的硬max变换器，利用链式思维（CoT）模拟图灵机。这使我们能够将构造转换为等效的softmax变换器，而无需先前方法所需的不现实的参数规模或激活精度。使用相同的技术，我们分析了最近提出的总结Co T范式，并展示其在模拟图灵机时更加高效，模型大小以空间界而非时间界缩放。我们通过在数独推理任务上验证我们的结果，并发现其比先前的高精度结果更符合可学习性。我们的代码可在https://github.com/moritzbroe/transformer-expressivity上获得。
英文摘要：
Existing expressivity results for transformers typically rely on hardmax attention, high precision, and other architectural modifications that disconnect them from the models used in practice. We bridge this gap by analyzing standard transformer decoders with softmax attention and rounding of activations and attention weights, while allowing depth and width to grow logarithmically with the context length. As an intermediate step, we construct hardmax transformers with ternary activations and well-separated attention scores that simulate Turing machines using Chain-of-Thought (CoT). This lets us convert the constructions to equivalent softmax transformers without the unrealistic parameter magnitudes or activation precision that prior approaches would require. Using the same technique, we analyze a recently proposed summarized CoT paradigm and show that it simulates Turing machines more efficiently, with model size scaling logarithmically in a space bound rather than a time bound. We empirically test predictions made by our results on a Sudoku reasoning task and find better alignment with learnability than for prior high-precision results. Our code is available at https://github.com/moritzbroe/transformer-expressivity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
