基于Transformer的模型在各种NLP、视觉和语音任务中取得了巨大成功。然而，Transformer的核心，即自注意力机制，在序列长度上具有二次时间和内存复杂度，这阻碍了基于Transformer的模型在长序列中的应用。已经提出了许多方法来缓解这个问题，例如稀疏注意力机制、低秩矩阵近似和可扩展核，以及自注意力的Token混合替代方案。我们提出了一种新颖的池化网络用于长序列中具有线性复杂度的Token混合。我们设计了多粒度池化和池化融合来捕获不同级别的上下文信息，并组合它们与Token的交互。在Long Range Arena基准上，PoNet显著优于Transformer并实现了有竞争力的准确率，同时在GPU上测量的所有序列长度上仅比最快的模型FNet略慢。我们还对PoNet的迁移学习能力进行了系统研究，观察到PoNet在GLUE基准上达到了BERT准确率的95.7%，相对FNet高出4.5%。全面的消融分析证明了所设计的多粒度池化和池化融合对于长序列中Token混合的有效性，以及所设计的预训练任务对于PoNet学习可迁移上下文语言表示的有效性。
英文摘要：
Transformer-based models have achieved great success in various NLP, vision, and speech tasks. However, the core of Transformer, the self-attention mechanism, has a quadratic time and memory complexity with respect to the sequence length, which hinders applications of Transformer-based models to long sequences. Many approaches have been proposed to mitigate this problem, such as sparse attention mechanisms, low-rank matrix approximations and scalable kernels, and token mixing alternatives to self-attention. We propose a novel Pooling Network (PoNet) for token mixing in long sequences with linear complexity. We design multi-granularity pooling and pooling fusion to capture different levels of contextual information and combine their interactions with tokens. On the Long Range Arena benchmark, PoNet significantly outperforms Transformer and achieves competitive accuracy, while being only slightly slower than the fastest model, FNet, across all sequence lengths measured on GPUs. We also conduct systematic studies on the transfer learning capability of PoNet and observe that PoNet achieves 95.7% of the accuracy of BERT on the GLUE benchmark, outperforming FNet by 4.5% relative. Comprehensive ablation analysis demonstrates effectiveness of the designed multi-granularity pooling and pooling fusion for token mixing in long sequences and efficacy of the designed pre-training tasks for PoNet to learn transferable contextualized language representations.
Enhancing Hyperspace Analogue to Language (HAL) Representations via Attention-Based Pooling for Text Classification
Undergraduate Students, Department of Computer and Automatic Control Engineering, Faculty of Mechanical and Electrical Engineering, Lattakia University（本科生，计算机与自动控制工程系，机械与电气工程学院，拉塔基亚大学）
Core Context Aware Transformers for Long Context Language Modeling
South China University of Technology（华南理工大学）；Peng Cheng Laboratory（鹏城实验室）；Pazhou Laboratory（琶洲实验室）；Harbin Institute of Technology（哈尔滨工业大学）；Key Laboratory of Big Data and Intelligent Robot, Ministry of Education（教育部大数据与智能机器人重点实验室）
Selecting What Matters: Semantic Compression-Guided Selective Pooling for Long-Context Embeddings
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；National University of Singapore（新加坡国立大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）
Interpreting token compositionality in LLMs: A robustness analysis
Department of Computer Science, University of Manchester（曼彻斯特大学计算机科学系）；Idiap Research Institute（Idiap研究机构）；National Biomarker Centre, CRUK-MI, Univ. of Manchester（国家生物标记中心，CRUK-MI，曼彻斯特大学）
ASSET: Adaptive Speech Segmentation via Reinforcement Learning
Johns Hopkins University（约翰斯·霍普金斯大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；HLTCOE, Johns Hopkins University（约翰斯·霍普金斯大学语言与语音处理中心）；NVIDIA（英伟达公司）
