面向赋予Transformer系统组合性能力的研究
Towards Equipping Transformer with the Ability of Systematic Compositionality
AI总结:
针对Transformer系统组合性泛化能力不足的问题,提出组合感知Transformer模型CAT与两种预训练任务,基于BERT实现多层版本,在组合任务上优于基线且对标准语言理解任务影响极小。
摘要:
语言生成能力与人类认知的关键因素之一是系统组合性能力,即理解由已见过的基元组合而成的未见过示例的能力。然而,近期研究证据表明,Transformer难以基于已见过的基元泛化到组合上下文场景中。为此,我们率先提出一种具备组合感知能力的Transformer,名为CAT(组合感知Transformer),并设计了两种新颖的预训练任务来提升系统组合性。我们基于应用极为广泛的BERT,初步实现了多层CAT的成功构建。实验结果表明,CAT在组合感知任务上的表现优于基线模型,且对标准语言理解任务的效果影响极小。
英文摘要:
One of the key factors in language productivity and human cognition is the ability of systematic compositionality, which refers to understanding composed unseen examples of seen primitives. However, recent evidence reveals that the Transformers have difficulty generalizing the composed context based on the seen primitives. To this end, we take the first step to propose a compositionality-aware Transformer called CAT and two novel pre-training tasks to facilitate systematic compositionality. We tentatively provide a successful implementation of a multi-layer CAT on the basis of the especially popular BERT. The experimental results demonstrate that CAT outperforms baselines on compositionality-aware tasks with minimal impact on the effectiveness on standardized language understanding tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。