arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.02060cs.LGcs.AIcs.SC
收藏

终止型可微树专家

Terminating Differentiable Tree Experts

IBM Research – Zurich(IBM研究院苏黎世分部);ETH Zürich(苏黎世联邦理工学院);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所)

Jonathan Thomm, Michael Hersche, Giacomo Camposampiero, Aleksandar Terzić, Bernhard Schölkopf, Abbas Rahimi

AI总结:

本文改进神经符号 Differentiable Tree Machine,利用 mixture of experts 使参数量不随计算步数增长,并提出无 oracle 的终止算法,让模型自动学习并收敛到最优步数。

摘要:

我们推进了最近提出的神经符号 Differentiable Tree Machine(可微树机器),该模型结合 transformers 与 Tensor Product Representations(张量积表示)来学习树操作。我们研究了其架构,并提出两个关键组件。首先,我们通过引入 mixture of experts(专家混合),移除了在每一步中都会使用的一系列不同 transformer 层。由此得到的 Differentiable Tree Experts(可微树专家)模型,在计算中无论步数多少都具有恒定数量的参数;相比之下,此前 Differentiable Tree Machine 中的方法会随步数线性增长。鉴于步数方面的这种灵活性,我们还提出了一种新的终止算法,使模型能够自动选择执行多少步。所得 Terminating Differentiable Tree Experts(终止型可微树专家)模型在没有 oracle 的情况下,能够缓慢地学习预测步数。它可以在保持模型学习能力的同时做到这一点,并收敛到最优步数。

英文摘要:

We advance the recently proposed neuro-symbolic Differentiable Tree Machine, which learns tree operations using a combination of transformers and Tensor Product Representations. We investigate the architecture and propose two key components. We first remove a series of different transformer layers that are used in every step by introducing a mixture of experts. This results in a Differentiable Tree Experts model with a constant number of parameters for any arbitrary number of steps in the computation, compared to the previous method in the Differentiable Tree Machine with a linear growth. Given this flexibility in the number of steps, we additionally propose a new termination algorithm to provide the model the power to choose how many steps to make automatically. The resulting Terminating Differentiable Tree Experts model sluggishly learns to predict the number of steps without an oracle. It can do so while maintaining the learning capabilities of the model, converging to the optimal amount of steps.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑