arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.14837cs.LGcs.AI
收藏

利用注意力机制:使用基于注意力的自编码器进行高效序列缩减

Harnessing Attention Mechanisms: Efficient Sequence Reduction using Attention-based Autoencoders

Centre for Artificial Intelligence Research, University of Agder(阿格德尔大学人工智能研究中心);Department of ICT, University of Agder(阿格德尔大学信息与通信技术系);Oslo Metropolitan University(奥斯都会大学)

Daniel Biermann, Fabrizio Palumbo, Morten Goodwin, Ole-Christoffer Granmo

AI总结:

本研究提出了一种基于注意力的序列长度缩减方法,并在自编码器模型中验证其有效性,发现将序列缩减至原长度的四分之一时仍能以约90%的准确率重现原始序列。

摘要:

许多机器学习模型将维度的操作作为驱动力,使模型能够识别和学习数据中的重要特征。对于序列数据,这种操作通常发生在 token 维度级别。尽管许多任务需要改变序列长度本身,但序列长度缩减的步骤通常出于必要性且在单步内完成。据我们所知,没有模型将序列长度缩减步骤作为调整模型性能的额外机会。事实上,序列长度操作作为一个整体似乎是一个被忽视的方向。在本研究中,我们引入了一种新颖的基于注意力的方法,允许直接操作序列长度。为了探索该方法的能力,我们将其应用于自编码器模型。该自编码器将输入序列缩减为潜在空间中的较小序列。然后,它旨在从这种缩减形式中重现原始序列。在此设置中,我们探索了该方法在不同输入和潜在序列长度下的缩减性能。我们能够证明,当将原始序列缩减至其原始大小的一半时,自编码器保留了所有重要信息。当缩减至低至其原始大小的四分之一时,自编码器仍能以约 90% 的准确率重现原始序列。

英文摘要:

Many machine learning models use the manipulation of dimensions as a driving force to enable models to identify and learn important features in data. In the case of sequential data this manipulation usually happens on the token dimension level. Despite the fact that many tasks require a change in sequence length itself, the step of sequence length reduction usually happens out of necessity and in a single step. As far as we are aware, no model uses the sequence length reduction step as an additional opportunity to tune the models performance. In fact, sequence length manipulation as a whole seems to be an overlooked direction. In this study we introduce a novel attention-based method that allows for the direct manipulation of sequence lengths. To explore the method's capabilities, we employ it in an autoencoder model. The autoencoder reduces the input sequence to a smaller sequence in latent space. It then aims to reproduce the original sequence from this reduced form. In this setting, we explore the methods reduction performance for different input and latent sequence lengths. We are able to show that the autoencoder retains all the significant information when reducing the original sequence to half its original size. When reducing down to as low as a quarter of its original size, the autoencoder is still able to reproduce the original sequence with an accuracy of around 90%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑