递归中的递归:面向可扩展长度泛化的两级嵌套递归
Recursion in Recursion: Two-Level Nested Recursion for Length Generalization with Scalability
AI总结:
该研究提出两级嵌套递归框架RIR,结合两类递归模型的优势,实现了ListOps任务90%以上的高长度泛化性能,同时可高效处理LRA长序列任务,语言任务表现优于Transformer且无需特殊初始化。
摘要:
二元平衡树递归向量神经网络(BBT-RvNNs)按照预设的平衡二叉树结构执行序列组合,因此其非线性递归深度仅为$\log_2 n$($n$为序列长度)。这种对数级缩放特性使BBT-RvNNs在长序列任务(如Long Range Arena,简称LRA)上具备高效性与可扩展性。然而,这种计算效率是有代价的——BBT-RvNNs无法解决ListOps这类简单算术任务。\n反过来看,能在ListOps(以及形式逻辑推理等其他结构敏感任务)上取得成功的RvNN(如Beam Tree RvNN),其计算成本通常甚至比循环神经网络(RNNs)还要高出数倍。\n本文提出了一种名为Recursion in Recursion(递归中的递归,简称RIR)的新型框架,旨在两者之间取得平衡,兼具两类模型的部分优势。在RIR中,我们采用一种两级嵌套递归的形式:外层递归是一个$k$叉平衡树模型,其单元函数由另一个递归模型(内层递归)实现。内层递归我们选择Beam Tree RvNNs(BT-RvNN),为了在RIR框架内适配BT-RvNNs,我们还提出了一种新颖的波束对齐策略。总体而言,这意味着RIR的总递归深度上界为$k \log_k n$。\n我们表现最佳的基于RIR的模型是首个在ListOps上实现高($\geq 90\\%$)长度泛化性能,同时具备足够可扩展性、能够在LRA的长序列输入上完成训练的模型。此外,在LRA语言任务的准确率方面,它无需任何特殊初始化即可与结构化状态空间模型(SSMs)表现相当,且大幅优于Transformer。另一方面,尽管SSMs在LRA上的表现能略微超过RIR,但它们(SSMs)无法在ListOps上实现长度泛化。我们的代码可在以下链接获取:https://github.com/JRC1995/BeamRecursionFamily/。
英文摘要:
Binary Balanced Tree RvNNs (BBT-RvNNs) enforce sequence composition according to a preset balanced binary tree structure. Thus, their non-linear recursion depth is just $\log_2 n$ ($n$ being the sequence length). Such logarithmic scaling makes BBT-RvNNs efficient and scalable on long sequence tasks such as Long Range Arena (LRA). However, such computational efficiency comes at a cost because BBT-RvNNs cannot solve simple arithmetic tasks like ListOps. On the flip side, RvNNs (e.g., Beam Tree RvNN) that do succeed on ListOps (and other structure-sensitive tasks like formal logical inference) are generally several times more expensive than even RNNs. In this paper, we introduce a novel framework -- Recursion in Recursion (RIR) to strike a balance between the two sides - getting some of the benefits from both worlds. In RIR, we use a form of two-level nested recursion - where the outer recursion is a $k$-ary balanced tree model with another recursive model (inner recursion) implementing its cell function. For the inner recursion, we choose Beam Tree RvNNs (BT-RvNN). To adjust BT-RvNNs within RIR we also propose a novel strategy of beam alignment. Overall, this entails that the total recursive depth in RIR is upper-bounded by $k \log_k n$. Our best RIR-based model is the first model that demonstrates high ($\geq 90\%$) length-generalization performance on ListOps while at the same time being scalable enough to be trainable on long sequence inputs from LRA. Moreover, in terms of accuracy in the LRA language tasks, it performs competitively with Structured State Space Models (SSMs) without any special initialization - outperforming Transformers by a large margin. On the other hand, while SSMs can marginally outperform RIR on LRA, they (SSMs) fail to length-generalize on ListOps. Our code is available at: \url{https://github.com/JRC1995/BeamRecursionFamily/}.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。