高效的Beam Tree递归方法
Efficient Beam Tree Recursion
AI总结:
针对BT-RvNN内存占用过高的问题,研究者通过解耦评分函数与递归单元函数将内存降低10-16倍,刷新ListOps性能纪录,并提出将其改造为序列上下文编码器的策略,拓展了RvNN的可扩展性与应用场景。
摘要:
Beam Tree递归神经网络(BT-RvNN,束树递归神经网络)是近期被提出的Gumbel Tree RvNN的简单扩展,已被证实在ListOps任务上实现了最优的长度泛化性能,同时在其他任务上保持了相当的表现。然而,尽管BT-RvNN并非同类模型中内存消耗最高的,其内存占用仍然过高。本文指出,BT-RvNN内存使用的主要瓶颈在于评分函数与递归单元函数的耦合。我们提出了消除该瓶颈的策略,进一步简化了其内存占用机制。总体而言,我们的策略不仅将BT-RvNN的内存使用量降低了10到16倍,还在ListOps任务上创造了新的最优性能,同时在其他任务上保持了相近的表现。此外,我们还提出了一种策略,利用BT-RvNN生成的诱导隐树节点表示,将BT-RvNN从形式为$f:\u211d^{n \ imes d} \ ightarrow \u211d^{d}$的句子编码器,转换为形式为$f:\u211d^{n \ imes d} \ ightarrow \u211d^{n \ imes d}$的序列上下文编码器。因此,我们的方案不仅为递归神经网络的进一步可扩展性开辟了道路,还规范了BT-RvNN作为深度学习工具包中另一基础组件的使用方式,使其可以轻松与Transformers、结构化状态空间模型等主流模型堆叠或对接。
英文摘要:
Beam Tree Recursive Neural Network (BT-RvNN) was recently proposed as a simple extension of Gumbel Tree RvNN and it was shown to achieve state-of-the-art length generalization performance in ListOps while maintaining comparable performance on other tasks. However, although not the worst in its kind, BT-RvNN can be still exorbitantly expensive in memory usage. In this paper, we identify the main bottleneck in BT-RvNN's memory usage to be the entanglement of the scorer function and the recursive cell function. We propose strategies to remove this bottleneck and further simplify its memory usage. Overall, our strategies not only reduce the memory usage of BT-RvNN by $10$-$16$ times but also create a new state-of-the-art in ListOps while maintaining similar performance in other tasks. In addition, we also propose a strategy to utilize the induced latent-tree node representations produced by BT-RvNN to turn BT-RvNN from a sentence encoder of the form $f:\mathbb{R}^{n \times d} \rightarrow \mathbb{R}^{d}$ into a sequence contextualizer of the form $f:\mathbb{R}^{n \times d} \rightarrow \mathbb{R}^{n \times d}$. Thus, our proposals not only open up a path for further scalability of RvNNs but also standardize a way to use BT-RvNNs as another building block in the deep learning toolkit that can be easily stacked or interfaced with other popular models such as Transformers and Structured State Space models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。