arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.18333cs.LGcs.CL
收藏

论序列模型中的“归纳偏置”

On the "Induction Bias" in Sequence Models

Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc(高通人工智能研究,由高通技术公司发起)

M. Reza Ebrahimi, Michaël Defferrard, Sunny Panchal, Roland Memisevic

AI总结:

通过大规模实验比较Transformer和RNN在状态跟踪任务上的数据效率,发现Transformer需要更多训练数据且难以跨长度共享权重,而RNN通过权重共享实现有效学习。

摘要:

尽管基于Transformer的语言模型在实际应用中取得了显著成功,但近期研究对其执行状态跟踪的能力提出了担忧。特别是,越来越多的文献通过分布外(OOD)泛化失败(如长度外推)来展示这一局限性。在本工作中,我们将注意力转向这些局限性的分布内影响。我们在多种监督机制下对Transformer和循环神经网络(RNN)的数据效率进行了大规模实验研究。我们发现,Transformer所需的训练数据量随状态空间大小和序列长度的增长远快于RNN。此外,我们分析了学习到的状态跟踪机制在不同序列长度上的共享程度。我们表明,Transformer在不同长度上的权重共享可以忽略甚至有害,表明它们孤立地学习长度特定的解决方案。相比之下,循环模型通过跨长度共享权重实现了有效的摊销学习,使得一个序列长度的数据能够提高其他长度上的性能。这些结果共同表明,即使训练和评估分布匹配,状态跟踪仍然是Transformer的一个基本挑战。

英文摘要:

Despite the remarkable practical success of transformer-based language models, recent work has raised concerns about their ability to perform state tracking. In particular, a growing body of literature has shown this limitation primarily through failures in out-of-distribution (OOD) generalization, such as length extrapolation. In this work, we shift attention to the in-distribution implications of these limitations. We conduct a large-scale experimental study of the data efficiency of transformers and recurrent neural networks (RNNs) across multiple supervision regimes. We find that the amount of training data required by transformers grows much more rapidly with state-space size and sequence length than for RNNs. Furthermore, we analyze the extent to which learned state-tracking mechanisms are shared across different sequence lengths. We show that transformers exhibit negligible or even detrimental weight sharing across lengths, indicating that they learn length-specific solutions in isolation. In contrast, recurrent models exhibit effective amortized learning by sharing weights across lengths, allowing data from one sequence length to improve performance on others. Together, these results demonstrate that state tracking remains a fundamental challenge for transformers, even when training and evaluation distributions match.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑