文本压缩辅助的Transformer编码
Text Compression-aided Transformer Encoding
摘要：
文本编码是自然语言处理（NLP）中最重要的步骤之一。在当前最先进的Transformer编码器中，自注意力机制已经做得很好，这为许多NLP任务的性能带来了显着的改进。虽然Transformer编码器可以有效地捕获一般信息，在其产生的表示，骨干信息，意思是输入文本的要点，是不是特别focusedon.In本文中，我们提出了显式和隐式的文本压缩方法，以提高Transformer编码和评估模型使用这种方法在几个典型的下游任务，严重依赖于编码。我们的显式文本压缩方法使用专用模型来压缩文本，而我们的隐式文本压缩方法只是在主模型中添加了一个额外的模块来处理文本压缩。我们提出了三种集成方式，即骨干源端融合，目标端融合，双方融合，集成的骨干信息到基于transformer的模型，为各种下游任务。我们对基准数据集的评估表明，与强基线相比，所提出的显式和隐式文本压缩方法改善了结果。因此，我们得出结论，当将编码与基线模型进行比较时，文本压缩有助于编码器学习更好的语言表示。
英文摘要：
Text encoding is one of the most important steps in Natural Language Processing (NLP). It has been done well by the self-attention mechanism in the current state-of-the-art Transformer encoder, which has brought about significant improvements in the performance of many NLP tasks. Though the Transformer encoder may effectively capture general information in its resulting representations, the backbone information, meaning the gist of the input text, is not specifically focused on. In this paper, we propose explicit and implicit text compression approaches to enhance the Transformer encoding and evaluate models using this approach on several typical downstream tasks that rely on the encoding heavily. Our explicit text compression approaches use dedicated models to compress text, while our implicit text compression approach simply adds an additional module to the main model to handle text compression. We propose three ways of integration, namely backbone source-side fusion, target-side fusion, and both-side fusion, to integrate the backbone information into Transformer-based models for various downstream tasks. Our evaluation on benchmark datasets shows that the proposed explicit and implicit text compression approaches improve results in comparison to strong baselines. We therefore conclude, when comparing the encodings to the baseline models, text compression helps the encoders to learn better language representations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
