arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.01107cs.LGcs.AIcs.DS
收藏

使用循环 Transformer 模拟图算法

Simulation of Graph Algorithms with Looped Transformers

University of Waterloo(滑铁卢大学)

Artur Back de Luca, Kimon Fountoulakis

AI总结:

本文从理论上证明带图交互注意力头的循环 transformer 能以不随图规模增长的参数量模拟多种经典图算法,并分析有限精度限制与恒定宽度图灵完备性。

摘要:

由于取得了富有前景的经验进展,利用神经网络执行图算法近来引起了广泛关注。这促使人们进一步理解神经网络如何复现针对关系数据的推理步骤。在本工作中,我们从理论角度研究 transformer 网络在图上模拟算法的能力。我们采用的架构是一种循环 transformer,它带有与图进行交互的额外注意力头。我们通过构造证明,该架构能够模拟 Dijkstra 最短路径、广度优先搜索与深度优先搜索、Kosaraju 强连通分量等单个算法,也能够同时模拟多种算法。网络中的参数量不会随输入图规模增加,这意味着这些网络可以在任意图上模拟上述算法。尽管具有这一性质,我们仍表明,由于有限精度,我们的解决方案在模拟方面存在一个限制。最后,我们展示了当利用这些额外注意力头时,可在恒定宽度下获得图灵完备性结果。

英文摘要:

The execution of graph algorithms using neural networks has recently attracted significant interest due to promising empirical progress. This motivates further understanding of how neural networks can replicate reasoning steps with relational data. In this work, we study the ability of transformer networks to simulate algorithms on graphs from a theoretical perspective. The architecture we use is a looped transformer with extra attention heads that interact with the graph. We prove by construction that this architecture can simulate individual algorithms such as Dijkstra's shortest path, Breadth- and Depth-First Search, and Kosaraju's strongly connected components, as well as multiple algorithms simultaneously. The number of parameters in the networks does not increase with the input graph size, which implies that the networks can simulate the above algorithms for any graph. Despite this property, we show a limit to simulation in our solution due to finite precision. Finally, we show a Turing Completeness result with constant width when the extra attention heads are utilized.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑