arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.09371cs.LGcs.AIcs.CL
收藏

Transformers 能实现长度泛化但不具鲁棒性

Transformers Can Achieve Length Generalization But Not Robustly

Google DeepMind(谷歌DeepMind);University of Toronto(多伦多大学)

Yongchao Zhou, Uri Alon, Xinyun Chen, Xuezhi Wang, Rishabh Agarwal, Denny Zhou

AI总结:

本文通过整数加法任务测试了Transformer的长度泛化能力,发现使用正确的数据格式与位置编码组合可实现2.5倍长度外推,但该能力受随机种子等因素影响仍显脆弱。

摘要:

长度泛化定义为从较短训练序列外推到较长测试序列的能力,是语言模型面临的一项重大挑战。即使使用大规模 Transformers 处理相对简单的任务,该问题依然存在。本文测试了 Transformer 在两个整数相加任务上的长度泛化能力。我们证明,长度泛化的成功与数据格式和位置编码类型密切相关。通过使用数据格式和位置编码的正确组合,我们首次表明标准 Transformers 可以外推到 2.5 倍输入长度的序列长度。然而,与分布内泛化不同,长度泛化仍然脆弱,受随机权重初始化和训练数据顺序等因素显著影响,导致不同随机种子间存在巨大方差。

英文摘要:

Length generalization, defined as the ability to extrapolate from shorter training sequences to longer test ones, is a significant challenge for language models. This issue persists even with large-scale Transformers handling relatively straightforward tasks. In this paper, we test the Transformer's ability of length generalization using the task of addition of two integers. We show that the success of length generalization is intricately linked to the data format and the type of position encoding. Using the right combination of data format and position encodings, we show for the first time that standard Transformers can extrapolate to a sequence length that is 2.5x the input length. Nevertheless, unlike in-distribution generalization, length generalization remains fragile, significantly influenced by factors like random weight initialization and training data order, leading to large variances across different random seeds.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑