arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.09512cs.CL
收藏

用于非自回归生成的注意力多层感知机

Attentive Multi-Layer Perceptron for Non-autoregressive Generation

Shanghai Jiao Tong University(上海交通大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The University of Hong Kong(香港大学)

Shuyang Jiang, Jun Zhang, Jiangtao Feng, Lin Zheng, Lingpeng Kong

AI总结:

本文提出注意力多层感知机(AMLP),通过输入自适应的投影实现线性复杂度,并与非自回归模型结合,在文本到语音合成和机器翻译上显著提升效率与性能。

摘要:

自回归(AR)生成因其有效性几乎主导了序列生成领域。近年来,非自回归(NAR)生成因其高效性和日益增长的效能而受到越来越多的关注。然而,其效率仍受限于序列长度的二次方复杂度,这阻碍了其扩展到长序列生成,且很少有工作致力于缓解这一问题。在本文中,我们提出了一种新颖的MLP变体——注意力多层感知机(AMLP),以生成具有线性和空间复杂度的时间生成模型。与具有静态和可学习投影矩阵的经典MLP不同,AMLP利用从输入中以注意力模式计算的自适应投影。样本感知的自适应投影使得序列中的标记之间能够进行通信,并建模查询空间与键空间之间的度量。此外,我们将AMLP与流行的NAR模型相结合,推导出一种具有线性和空间复杂度的高效NAR-AMLP架构。实验结果表明,这种结合架构在文本到语音合成和机器翻译上显著超越了竞争性的高效NAR模型。我们还通过广泛的消融实验分别测试了AMLP的自注意力和交叉注意力能力,发现它们与其他高效模型相比表现相当甚至更优。效率分析进一步表明,AMLP在长序列上相比原始非自回归模型极大地降低了内存成本。

英文摘要:

Autoregressive~(AR) generation almost dominates sequence generation for its efficacy. Recently, non-autoregressive~(NAR) generation gains increasing popularity for its efficiency and growing efficacy. However, its efficiency is still bottlenecked by quadratic complexity in sequence lengths, which is prohibitive for scaling to long sequence generation and few works have been done to mitigate this problem. In this paper, we propose a novel MLP variant, \textbf{A}ttentive \textbf{M}ulti-\textbf{L}ayer \textbf{P}erceptron~(AMLP), to produce a generation model with linear time and space complexity. Different from classic MLP with static and learnable projection matrices, AMLP leverages adaptive projections computed from inputs in an attentive mode. The sample-aware adaptive projections enable communications among tokens in a sequence, and model the measurement between the query and key space. Furthermore, we marry AMLP with popular NAR models, deriving a highly efficient NAR-AMLP architecture with linear time and space complexity. Empirical results show that such marriage architecture surpasses competitive efficient NAR models, by a significant margin on text-to-speech synthesis and machine translation. We also test AMLP's self- and cross-attention ability separately with extensive ablation experiments, and find them comparable or even superior to the other efficient models. The efficiency analysis further shows that AMLP extremely reduces the memory cost against vanilla non-autoregressive models for long sequences.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑