arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.05982cs.CLcs.AI
收藏

基于非纠缠序列到序列学习的真实世界合成泛化

Real-World Compositional Generalization with Disentangled Sequence-to-Sequence Learning

Institute for Language, Cognition and Computation(语言、认知与计算研究所);School of Informatics, University of Edinburgh(爱丁堡大学信息学院)

Hao Zheng, Mirella Lapata

摘要:

成分泛化是人类语言学习的一个基本机制,目前的神经网络正在努力解决这一问题。最近提出的解纠缠序列到序列模型(Dangle)通过为每个解码步骤学习专门的编码,显示出很有前途的泛化能力。我们对这个模型进行了两个关键的修改,鼓励更多的解纠缠表示,并提高其计算和内存效率,使我们能够在更现实的环境中处理合成泛化。具体来说,我们不是在每个时间步自适应地重新编码源键和值,而是解开它们的表示,只在某个时间间隔周期性地重新编码键。我们的新架构在现有的任务和数据集上带来了更好的泛化性能,以及一个新的机器翻译基准,我们通过检测与训练集相关的自然发生的组成模式来创建该基准。我们表明,这种方法比人工挑战更好地模拟现实世界的要求。

英文摘要:

Compositional generalization is a basic mechanism in human language learning, which current neural networks struggle with. A recently proposed Disentangled sequence-to-sequence model (Dangle) shows promising generalization capability by learning specialized encodings for each decoding step. We introduce two key modifications to this model which encourage more disentangled representations and improve its compute and memory efficiency, allowing us to tackle compositional generalization in a more realistic setting. Specifically, instead of adaptively re-encoding source keys and values at each time step, we disentangle their representations and only re-encode keys periodically, at some interval. Our new architecture leads to better generalization performance across existing tasks and datasets, and a new machine translation benchmark which we create by detecting naturally occurring compositional patterns in relation to a training set. We show this methodology better emulates real-world requirements than artificial challenges.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑