arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.19150cs.CL
收藏

自回归语言建模中决策树的能力研究

On the Power of Decision Trees in Auto-Regressive Language Modeling

Massachusetts Institute of Technology(麻省理工学院);Texas A&M University(德克萨斯农工大学);Harvard University(哈佛大学)

Yulu Gan, Tomer Galanti, Tomaso Poggio, Eran Malach

AI总结:

本文首次将自回归决策树(ARDTs)用于语言建模,从理论上证明其借助思维链可模拟自动机、图灵机和稀疏电路,并通过实验表明其在文本生成和复杂推理任务上具备与小型Transformer相当的能力,为语言模型架构多样化提供新方向。

摘要:

自回归决策树(ARDTs)最初为处理时间序列数据而提出,但尚未在语言建模领域得到探索。本文深入研究了ARDTs在这一新背景下的理论与实际应用。我们从理论上证明,ARDTs可以通过利用“思维链”计算来执行复杂函数,例如模拟自动机、图灵机和稀疏电路。我们的分析给出了ARDTs在规模、深度和计算效率方面的界限,突显了其令人惊讶的计算能力。在实证方面,我们在简单的语言生成任务上训练ARDTs,结果表明它们能够学习生成连贯且语法正确的文本,其水平可与较小的Transformer模型相媲美。此外,我们还展示了ARDTs可以构建在Transformer表示之上,用于解决复杂推理任务。本研究揭示了ARDTs独特的计算能力,旨在拓宽语言模型开发中的架构多样性。

英文摘要:

Originally proposed for handling time series data, Auto-regressive Decision Trees (ARDTs) have not yet been explored for language modeling. This paper delves into both the theoretical and practical applications of ARDTs in this new context. We theoretically demonstrate that ARDTs can compute complex functions, such as simulating automata, Turing machines, and sparse circuits, by leveraging "chain-of-thought" computations. Our analysis provides bounds on the size, depth, and computational efficiency of ARDTs, highlighting their surprising computational power. Empirically, we train ARDTs on simple language generation tasks, showing that they can learn to generate coherent and grammatically correct text on par with a smaller Transformer model. Additionally, we show that ARDTs can be used on top of transformer representations to solve complex reasoning tasks. This research reveals the unique computational abilities of ARDTs, aiming to broaden the architectural diversity in language model development.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑