arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.18512cs.LGcs.AI
收藏

通过图算法理解Transformer的推理能力

Understanding Transformer Reasoning Capabilities via Graph Algorithms

Google Research(谷歌研究院);Columbia University(哥伦比亚大学);Google DeepMind(谷歌DeepMind)

Clayton Sanford, Bahare Fatemi, Ethan Hall, Anton Tsitsulin, Mehran Kazemi, Jonathan Halcrow, Bryan Perozzi, Vahab Mirrokni

AI总结:

本文通过引入表示层级,从深度、宽度和额外标记维度分析Transformer在9个算法推理问题上的能力,证明了对数深度和小嵌入单层的适用性,并在GraphQA基准上验证其优于图神经网络。

摘要:

哪些Transformer扩展机制能够完美解决不同类别的算法问题?尽管基于Transformer的神经网络取得了巨大的实证进展,但对其在现实参数机制下的算法推理能力仍缺乏理论理解。我们从网络深度、宽度以及用于算法执行的额外标记数量的角度研究了这个问题。我们提出了一种新颖的表示层级,将9个算法推理问题划分为可由不同现实参数扩展机制下的Transformer解决的类别。我们证明了对数深度对于图连通性等任务是必要且充分的,而具有小嵌入维度的单层Transformer可以解决上下文检索任务。我们还使用GraphQA基准为我们的理论分析提供了充足的实证证据。这些结果表明,Transformer在许多图推理任务中表现出色,甚至优于专门的图神经网络。

英文摘要:

Which transformer scaling regimes are able to perfectly solve different classes of algorithmic problems? While tremendous empirical advances have been attained by transformer-based neural networks, a theoretical understanding of their algorithmic reasoning capabilities in realistic parameter regimes is lacking. We investigate this question in terms of the network's depth, width, and number of extra tokens for algorithm execution. Our novel representational hierarchy separates 9 algorithmic reasoning problems into classes solvable by transformers in different realistic parameter scaling regimes. We prove that logarithmic depth is necessary and sufficient for tasks like graph connectivity, while single-layer transformers with small embedding dimensions can solve contextual retrieval tasks. We also support our theoretical analysis with ample empirical evidence using the GraphQA benchmark. These results show that transformers excel at many graph reasoning tasks, even outperforming specialized graph neural networks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑