Transformer 遇见神经算法推理器
Transformers meet Neural Algorithmic Reasoners
AI总结：
本文提出混合架构 TransNAR，通过两阶段训练让 Transformer 交叉注意 NAR 的节点嵌入，从而增强语言模型的算法推理能力。
摘要：
Transformer 以其简单而有效的架构彻底改变了机器学习。在来自互联网的海量文本数据集上预训练 Transformer，为自然语言理解（NLU）任务带来了无与伦比的泛化能力。然而，当面对算法形式的推理任务时，这类语言模型仍然较为脆弱，因为其中的计算必须精确且稳健。为解决这一局限，我们提出一种新颖方法，将 Transformer 的语言理解能力与基于图神经网络（GNN）的神经算法推理器（NAR）的稳健性相结合。当算法任务以图形式指定时，此类 NAR 已被证明是有效的通用求解器。为使其嵌入可被 Transformer 使用，我们提出一种混合架构及两阶段训练流程，使语言模型中的 token 能够交叉注意来自 NAR 的节点嵌入。我们在 CLRS-Text（CLRS-30 基准的文本版本）上评估所得 TransNAR 模型，结果表明，在分布内和分布外的算法推理方面，其相较仅使用 Transformer 的模型均取得显著提升。
英文摘要：
Transformers have revolutionized machine learning with their simple yet effective architecture. Pre-training Transformers on massive text datasets from the Internet has led to unmatched generalization for natural language understanding (NLU) tasks. However, such language models remain fragile when tasked with algorithmic forms of reasoning, where computations must be precise and robust. To address this limitation, we propose a novel approach that combines the Transformer's language understanding with the robustness of graph neural network (GNN)-based neural algorithmic reasoners (NARs). Such NARs proved effective as generic solvers for algorithmic tasks, when specified in graph form. To make their embeddings accessible to a Transformer, we propose a hybrid architecture with a two-phase training procedure, allowing the tokens in the language model to cross-attend to the node embeddings from the NAR. We evaluate our resulting TransNAR model on CLRS-Text, the text-based version of the CLRS-30 benchmark, and demonstrate significant gains over Transformer-only models for algorithmic reasoning, both in and out of distribution.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
