通过递归组合的多粒度表示增强Transformer
Augmenting Transformers with Recursively Composed Multi-grained Representations
AI总结：
本研究提出递归组合增强Transformer模型ReCAT，通过新增的上下文内外层实现跨度内外信息交互，生成多粒度表示，在多项任务上性能优于基线，且诱导的句法结构与人工标注一致性高、可解释性强。
摘要：
我们提出了ReCAT（递归组合增强Transformer），这是一种能够显式建模原始文本层级句法结构的Transformer，在训练和推理阶段均无需依赖金标句法树。该领域现有研究限制数据遵循层级树结构，因此缺乏跨度间的信息交互。为解决这一问题，我们提出了一种新颖的上下文内外（CIO）层，通过自底向上和自顶向下的传播过程学习跨度的上下文化表示：其中自底向上传播通过组合低层跨度的表示来构建高层跨度的表示，而自顶向下传播则融合跨度内部与外部的信息。通过在Transformer的嵌入层和注意力层之间堆叠多个CIO层，ReCAT模型能够同时实现深度的跨度内交互和深度的跨度间交互，从而生成与其他跨度完全上下文化关联的多粒度表示。此外，CIO层可与Transformer联合预训练，使得ReCAT同时具备可扩展性、强劲性能和可解释性。我们在多种句子级和跨度级任务上开展了实验。评估结果表明，ReCAT在所有跨度级任务上的表现均显著优于普通Transformer模型，在自然语言推理任务上也优于将递归网络与Transformer结合的基线模型。更值得注意的是，ReCAT诱导出的层级结构与人工标注的句法树表现出高度一致性，证明了CIO层带来的良好可解释性。
英文摘要：
We present ReCAT, a recursive composition augmented Transformer that is able to explicitly model hierarchical syntactic structures of raw texts without relying on gold trees during both learning and inference. Existing research along this line restricts data to follow a hierarchical tree structure and thus lacks inter-span communications. To overcome the problem, we propose a novel contextual inside-outside (CIO) layer that learns contextualized representations of spans through bottom-up and top-down passes, where a bottom-up pass forms representations of high-level spans by composing low-level spans, while a top-down pass combines information inside and outside a span. By stacking several CIO layers between the embedding layer and the attention layers in Transformer, the ReCAT model can perform both deep intra-span and deep inter-span interactions, and thus generate multi-grained representations fully contextualized with other spans. Moreover, the CIO layers can be jointly pre-trained with Transformers, making ReCAT enjoy scaling ability, strong performance, and interpretability at the same time. We conduct experiments on various sentence-level and span-level tasks. Evaluation results indicate that ReCAT can significantly outperform vanilla Transformer models on all span-level tasks and baselines that combine recursive networks with Transformers on natural language inference tasks. More interestingly, the hierarchical structures induced by ReCAT exhibit strong consistency with human-annotated syntactic trees, indicating good interpretability brought by the CIO layers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
