用Treeformer构建树结构
Forming Trees with Treeformers
AI总结：
本文提出受CKY算法启发的通用编码器模块Treeformer，通过将层次结构融入Transformer，显著提升组合泛化及机器翻译、摘要和自然语言理解等下游任务性能。
摘要：
众所周知，人类语言呈现出嵌套的层次结构，使我们能够由较小的成分组成复杂的句子。然而，许多最先进的神经网络模型（如Transformer）在其架构中没有显式的层次结构——也就是说，它们没有面向层次结构的归纳偏置。此外，已知Transformer在需要此类结构的组合泛化任务上表现不佳。在本文中，我们介绍了Treeformer，这是一种受CKY算法启发的通用编码器模块，它学习组合算子和池化函数，为短语和句子构建层次化编码。我们的大量实验证明了将层次结构融入Transformer的益处，并显示出在组合泛化以及机器翻译、抽象式摘要和各种自然语言理解等下游任务上的显著改进。
英文摘要：
Human language is known to exhibit a nested, hierarchical structure, allowing us to form complex sentences out of smaller pieces. However, many state-of-the-art neural networks models such as Transformers have no explicit hierarchical structure in its architecture -- that is, they don't have an inductive bias toward hierarchical structure. Additionally, Transformers are known to perform poorly on compositional generalization tasks which require such structures. In this paper, we introduce Treeformer, a general-purpose encoder module inspired by the CKY algorithm which learns a composition operator and pooling function to construct hierarchical encodings for phrases and sentences. Our extensive experiments demonstrate the benefits of incorporating hierarchical structure into the Transformer and show significant improvements in compositional generalization as well as in downstream tasks such as machine translation, abstractive summarization, and various natural language understanding tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
