arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.22690cs.LGcs.AI
收藏

Transformer 是否具备周期性泛化能力?

Do Transformers Have the Ability for Periodicity Generalization?

School of Computer Science, Peking University(北京大学计算机科学学院);College of AI, Tsinghua University(清华大学人工智能学院);School of Science and Engineering, The Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳)科学与工程学院)

Huanyu Liu, Ge Li, Yihong Dong, Sihan Wu, Peixu Wang, Sihao Cheng, Taozhi Chen, Kechi Zhang, Hao Zhu, Tongxuan Liu

AI总结:

本文研究了 Transformer 在周期性泛化方面的局限性,通过引入抽象代数和推理视角,揭示了模型在处理复合周期性任务时的不足,并提出了可控生成基准 Coper。

摘要:

基于 Transformer 的大型语言模型(LLMs)在多种任务上表现出强大的性能。然而,当前模型在分布外(OOD)泛化方面仍与人类相比存在显著差距。我们通过周期性,一种基本的 OOD 场景,来研究这一差距。周期性捕捉了在变化中保持不变的特性。周期性泛化代表了模型从训练数据中提取周期性模式并泛化到 OOD 场景的能力。我们从抽象代数和推理的角度引入了周期性的统一解释,包括单周期性和复合周期性,以解释为什么 Transformer 在泛化周期性方面存在困难。然后我们构建了 Coper 关于复合周期性,一个具有两个 OOD 设置的可控生成基准,Hollow 和 Extrapolation。实验表明,Transformer 的周期性泛化能力有限,模型在训练期间可以记忆周期性数据,但无法泛化到未见的复合周期性。我们发布了源代码以支持未来研究。

英文摘要:

Large language models (LLMs) based on the Transformer have demonstrated strong performance across diverse tasks. However, current models still exhibit substantial limitations in out-of-distribution (OOD) generalization compared with humans. We investigate this gap through periodicity, one of the basic OOD scenarios. Periodicity captures invariance amid variation. Periodicity generalization represents a model's ability to extract periodic patterns from training data and generalize to OOD scenarios. We introduce a unified interpretation of periodicity from the perspective of abstract algebra and reasoning, including both single and composite periodicity, to explain why Transformers struggle to generalize periodicity. Then we construct Coper about composite periodicity, a controllable generative benchmark with two OOD settings, Hollow and Extrapolation. Experiments reveal that periodicity generalization in Transformers is limited, where models can memorize periodic data during training, but cannot generalize to unseen composite periodicity. We release the source code to support future research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑