arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.15607cs.LG
收藏

非线性 Transformer 如何在上下文学习中学习与泛化?

How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?

Rensselaer Polytechnic Institute(伦斯勒理工学院);IBM Thomas J. Watson Research Center(IBM托马斯·J·沃森研究中心)

Hongkang Li, Meng Wang, Songtao Lu, Xiaodong Cui, Pin-Yu Chen

AI总结:

本文首次理论分析含非线性自注意力与 MLP 的 Transformer 的 ICL 训练动态、跨任务泛化、组件作用及幅度剪枝影响,并以数值实验验证。

摘要:

基于 Transformer 的大型语言模型已展现出令人瞩目的上下文学习(ICL)能力:预训练模型只需用来自某任务的若干输入-输出示例来增强查询,即可在不进行微调的情况下处理新任务。尽管在实证上取得了成功,但由于 Transformer 中非线性自注意力和非线性激活所产生的非凸训练问题在分析上存在技术挑战,关于如何训练 Transformer 以实现 ICL 的机制以及相应 ICL 能力,人们仍知之甚少。据我们所知,本文首次对具有非线性自注意力和非线性 MLP 的 Transformer 的训练动态,以及所得模型的 ICL 泛化能力进行了理论分析。围绕一组二分类任务,我们使用这些任务中一个子集的数据训练 Transformer,并量化多种因素对其余未见任务上 ICL 泛化性能的影响,实验分别在有无数据分布偏移的条件下进行。我们还分析了所学 Transformer 中不同组件如何贡献于 ICL 性能。此外,我们首次对模型剪枝如何影响 ICL 性能给出理论分析,并证明适当的基于幅度的剪枝能够在降低推理成本的同时,对 ICL 产生极小影响。这些理论发现通过数值实验得到了验证。

英文摘要:

Transformer-based large language models have displayed impressive in-context learning capabilities, where a pre-trained model can handle new tasks without fine-tuning by simply augmenting the query with some input-output examples from that task. Despite the empirical success, the mechanics of how to train a Transformer to achieve ICL and the corresponding ICL capacity is mostly elusive due to the technical challenges of analyzing the nonconvex training problems resulting from the nonlinear self-attention and nonlinear activation in Transformers. To the best of our knowledge, this paper provides the first theoretical analysis of the training dynamics of Transformers with nonlinear self-attention and nonlinear MLP, together with the ICL generalization capability of the resulting model. Focusing on a group of binary classification tasks, we train Transformers using data from a subset of these tasks and quantify the impact of various factors on the ICL generalization performance on the remaining unseen tasks with and without data distribution shifts. We also analyze how different components in the learned Transformers contribute to the ICL performance. Furthermore, we provide the first theoretical analysis of how model pruning affects ICL performance and prove that proper magnitude-based pruning can have a minimal impact on ICL while reducing inference costs. These theoretical findings are justified through numerical experiments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑