DCT-Former:高效的离散余弦变换自注意力
DCT-Former: Efficient Self-Attention with Discrete Cosine Transform
摘要:
自其引入以来,Transformer架构成为自然语言处理和最近计算机视觉应用的主导架构。这一系列“全注意”架构的内在限制来自点积注意力的计算,其在存储器消耗和操作数量方面都增长为$O(n^2)$,其中$n$代表输入序列长度,从而限制了需要对非常长的序列建模的应用。到目前为止,文献中已经提出了几种方法来缓解这个问题,并取得了不同程度的成功。我们的想法从有损数据压缩(如JPEG算法)的世界中获得灵感,通过利用离散余弦变换的属性来获得注意力模块的近似值。大量的实验表明,我们的方法在相同的性能下占用更少的内存,同时也大大减少了推理时间。这使得它特别适合嵌入式平台上的实时环境。此外,我们假设我们的研究结果可以作为更广泛的深度神经模型家族的起点,这些模型具有更少的内存占用。该实现将在https://github.com/cscribano/DCT-Former-Public上公开提供
英文摘要:
Since their introduction the Trasformer architectures emerged as the dominating architectures for both natural language processing and, more recently, computer vision applications. An intrinsic limitation of this family of "fully-attentive" architectures arises from the computation of the dot-product attention, which grows both in memory consumption and number of operations as $O(n^2)$ where $n$ stands for the input sequence length, thus limiting the applications that require modeling very long sequences. Several approaches have been proposed so far in the literature to mitigate this issue, with varying degrees of success. Our idea takes inspiration from the world of lossy data compression (such as the JPEG algorithm) to derive an approximation of the attention module by leveraging the properties of the Discrete Cosine Transform. An extensive section of experiments shows that our method takes up less memory for the same performance, while also drastically reducing inference time. This makes it particularly suitable in real-time contexts on embedded platforms. Moreover, we assume that the results of our research might serve as a starting point for a broader family of deep neural models with reduced memory footprint. The implementation will be made publicly available at https://github.com/cscribano/DCT-Former-Public
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。