通过预训练执行句法变换来强化结构性归纳偏置
Strengthening Structural Inductive Biases by Pre-training to Perform Syntactic Transformations
AI总结：
本文提出通过中间预训练执行依存树句法变换来强化Transformer的结构性归纳偏置，实验证明该方法能提升句法任务的少样本学习和语义解析的结构泛化能力。
摘要：
模型需要适当的归纳偏置，才能从少量数据中有效学习并在训练分布之外进行系统性泛化。尽管Transformer高度通用且功能强大，但在seq2seq任务中，尤其是涉及句法变换的任务（例如将主动语态转换为被动语态或语义解析）中，它们仍可受益于增强的结构性归纳偏置。在本文中，我们提出通过对Transformer进行中间预训练，使其在给定变换描述的情况下执行自动生成的依存树的句法变换，从而强化其结构性归纳偏置。我们的实验证实，这有助于对诸如组块分析等句法任务的少样本学习，并且还改善了语义解析的结构泛化能力。我们的分析表明，中间预训练使注意力头能够跟踪需要对哪个词元应用哪种句法变换，并且该模型可以在下游任务中利用这些注意力头。
英文摘要：
Models need appropriate inductive biases to effectively learn from small amounts of data and generalize systematically outside of the training distribution. While Transformers are highly versatile and powerful, they can still benefit from enhanced structural inductive biases for seq2seq tasks, especially those involving syntactic transformations, such as converting active to passive voice or semantic parsing. In this paper, we propose to strengthen the structural inductive bias of a Transformer by intermediate pre-training to perform synthetically generated syntactic transformations of dependency trees given a description of the transformation. Our experiments confirm that this helps with few-shot learning of syntactic tasks such as chunking, and also improves structural generalization for semantic parsing. Our analysis shows that the intermediate pre-training leads to attention heads that keep track of which syntactic transformation needs to be applied to which token, and that the model can leverage these attention heads on downstream tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
