arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.00726cs.LGcs.AIstat.ML
收藏

通过任务提示提升Transformer的长度泛化能力

Improving Length-Generalization in Transformers via Task Hinting

Google Research(谷歌研究院);Carnegie Mellon University(卡内基梅隆大学)

Pranjal Awasthi, Anupam Gupta

AI总结:

针对Transformer的长度泛化难题,提出任务提示方法,通过同步训练相关辅助任务,结合多任务框架、机制分析与长度依赖参数优化,大幅提升排序等任务的长序列性能,拓展了技术适用场景。

摘要:

近年来研究发现,Transformer在某些类型的推理和算术任务上存在长度泛化问题。具体而言,在特定长度(例如5位数)以内的任务(比如加法)上训练的Transformer模型,在应用于同一问题的更长实例时,性能会急剧下降。本文提出了一种基于任务提示的方法来解决长度泛化问题。我们的核心思路是:在针对特定任务数据训练模型的同时,同步训练模型解决一个更简单但相关的辅助任务,将有助于提升长度泛化能力。\n我们以经典的排序问题作为典型示例来评估所提方法。我们设计了一个多任务训练框架,并证明任务提示能显著提升长度泛化能力。针对排序任务的实验表明,在序列长度不超过20的数据集上训练模型时,通过任务提示,模型在长度为100的序列上的测试准确率从标准训练的不足1%提升至92%以上。\n我们的研究揭示了长度泛化的若干有趣特性。我们观察到,尽管许多辅助任务从先验角度看似乎很合理,但它们在提升长度泛化方面的效果差异极大。我们进一步使用基于探针和可视化的技术来理解模型执行任务的内部机制,并提出了一个与观测到的模型学习行为一致的理论构造。基于该构造,我们证明在训练过程中引入少量长度依赖参数,可以进一步提升模型在未见过的长度上的性能。最后,我们还验证了基于任务提示的方法在排序任务之外的有效性,为这些技术应用于更广泛的场景带来了希望。

英文摘要:

It has been observed in recent years that transformers have problems with length generalization for certain types of reasoning and arithmetic tasks. In particular, the performance of a transformer model trained on tasks (say addition) up to a certain length (e.g., 5 digit numbers) drops sharply when applied to longer instances of the same problem. This work proposes an approach based on task hinting towards addressing length generalization. Our key idea is that while training the model on task-specific data, it is helpful to simultaneously train the model to solve a simpler but related auxiliary task as well. We study the classical sorting problem as a canonical example to evaluate our approach. We design a multitask training framework and show that task hinting significantly improve length generalization. For sorting we show that it is possible to train models on data consisting of sequences having length at most $20$, and improve the test accuracy on sequences of length $100$ from less than 1% (for standard training) to more than 92% (via task hinting). Our study uncovers several interesting aspects of length generalization. We observe that while several auxiliary tasks may seem natural a priori, their effectiveness in improving length generalization differs dramatically. We further use probing and visualization-based techniques to understand the internal mechanisms via which the model performs the task, and propose a theoretical construction consistent with the observed learning behaviors of the model. Based on our construction, we show that introducing a small number of length dependent parameters into the training procedure can further boost the performance on unseen lengths. Finally, we also show the efficacy of our task hinting based approach beyond sorting, giving hope that these techniques will be applicable in broader contexts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑