arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1911.04070cs.CLcs.LG
收藏

BP-Transformer:基于二进制划分的远程上下文建模

BP-Transformer: Modelling Long-Range Context via Binary Partitioning

AWS Shanghai AI Lab(AWS上海人工智能实验室);Fudan University(复旦大学);New York University Shanghai(上海纽约大学)

Zihao Ye, Qipeng Guo, Quan Gan, Xipeng Qiu, Zheng Zhang

摘要:

Transformer模型在许多自然语言处理任务上获得了广泛的成功。然而,自我注意的二次复杂性限制了它在长文本中的应用。本文采用一种基于二进制划分的多尺度由细到粗的注意力机制,提出了BP-Transformer(简称BPT)。BPT产生$O(k\cdot n\log (n/k))$连接,其中$k$是控制注意力密度的超参数。BPT在计算复杂度和模型容量之间有很好的平衡。一系列的文本分类、机器翻译和语言建模实验表明,BPT模型在长文本分类方面的性能优于以往的自注意模型。我们的代码,超参数和用于稀疏注意力的CUDA内核可在PyTorch中使用。

英文摘要:

The Transformer model is widely successful on many natural language processing tasks. However, the quadratic complexity of self-attention limit its application on long text. In this paper, adopting a fine-to-coarse attention mechanism on multi-scale spans via binary partitioning (BP), we propose BP-Transformer (BPT for short). BPT yields $O(k\cdot n\log (n/k))$ connections where $k$ is a hyperparameter to control the density of attention. BPT has a good balance between computation complexity and model capacity. A series of experiments on text classification, machine translation and language modeling shows BPT has a superior performance for long text than previous self-attention models. Our code, hyperparameters and CUDA kernels for sparse attention are available in PyTorch.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑