arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.02625cs.LGcs.AIcs.CL
收藏

利用多时间视角增强Transformer RNNs

Enhancing Transformer RNNs with Multiple Temporal Perspectives

University of Arizona(亚利桑那大学);MultiversX;Lucian Blaga University of Sibiu(锡比乌卢西恩·布拉加大学)

Razvan-Gabriel Dumitru, Darius Peteleaza, Mihai Surdeanu

AI总结:

本研究提出多时间视角方法并融入RWKV架构,以极少量参数增量和微调开销解决单隐藏状态存历史信息的问题,保持推理线性复杂度,在多个基准上实现性能提升且相关资源开源。

摘要:

我们提出了多时间视角的概念,这是一种适用于循环神经网络(RNN)架构的新方法,用于增强其对序列数据的理解。该方法包括维护先前遇到文本的多种时间视图,显著丰富了语言模型解读上下文的能力。为了证明该方法的有效性,我们将其融入Receptance Weighted Key Value(RWKV, receptance加权键值)架构,解决其在单个隐藏状态中保留所有历史信息的固有挑战。值得注意的是,实现这一改进仅需极少的参数增量——甚至仅为原始参数数量的0.04%。此外,多时间视角所需的额外参数只需极小的计算开销即可微调,无需完整的预训练。所得模型在提示推理期间保持线性计算复杂度,确保在各种序列长度下的效率一致性。我们研究中的实证结果和消融实验验证了该方法的有效性,展现出在多个基准上的性能提升。代码、模型权重和数据集已在https://github.com/RazvanDu/TemporalRNNs开源。

英文摘要:

We introduce the concept of multiple temporal perspectives, a novel approach applicable to Recurrent Neural Network (RNN) architectures for enhancing their understanding of sequential data. This method involves maintaining diverse temporal views of previously encountered text, significantly enriching the language models' capacity to interpret context. To show the efficacy of this approach, we incorporate it into the Receptance Weighted Key Value (RWKV) architecture, addressing its inherent challenge of retaining all historical information within a single hidden state. Notably, this improvement is achieved with a minimal increase in the number of parameters --even as little as $0.04\%$ of the original number of parameters. Further, the additional parameters necessary for the multiple temporal perspectives are fine-tuned with minimal computational overhead, avoiding the need for a full pre-training. The resulting model maintains linear computational complexity during prompt inference, ensuring consistent efficiency across various sequence lengths. The empirical results and ablation studies included in our research validate the effectiveness of our approach, showcasing improved performance across multiple benchmarks. The code, model weights and datasets are open-sourced at: https://github.com/RazvanDu/TemporalRNNs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑