具有动态Token池化的高效Transformer
Efficient Transformers with Dynamic Token Pooling
AI总结：
提出动态池化机制，通过自回归预测段边界来减少Transformer中间层序列长度，在相同计算预算下比vanilla Transformer和固定长度池化更快更准确。
摘要：
Transformer在语言建模中表现无与伦比，但在内存和时间复杂度方面仍效率低下。一种可能的补救措施是通过池化固定长度的token段来减少中间层的序列长度。然而，自然意义单元（如词或短语）显示出不同的大小。为解决这种不匹配，我们为语言模型配备了动态池化机制，以自回归方式预测段边界。我们比较了几种推断边界的方法，包括通过随机重参数化的端到端学习、监督学习（基于子词分词器的分割或条件熵的峰值），以及语言动机边界。我们在来自多个数据集和形态多样语言的文本上进行字符级评估。结果表明，联合分割和建模语言的动态池化在相同计算预算下比vanilla Transformer和固定长度池化更快且更准确。
英文摘要：
Transformers achieve unrivalled performance in modelling language, but remain inefficient in terms of memory and time complexity. A possible remedy is to reduce the sequence length in the intermediate layers by pooling fixed-length segments of tokens. Nevertheless, natural units of meaning, such as words or phrases, display varying sizes. To address this mismatch, we equip language models with a dynamic-pooling mechanism, which predicts segment boundaries in an autoregressive fashion. We compare several methods to infer boundaries, including end-to-end learning through stochastic re-parameterisation, supervised learning (based on segmentations from subword tokenizers or spikes in conditional entropy), as well as linguistically motivated boundaries. We perform character-level evaluation on texts from multiple datasets and morphologically diverse languages. The results demonstrate that dynamic pooling, which jointly segments and models language, is both faster and more accurate than vanilla Transformers and fixed-length pooling within the same computational budget.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
