arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.02199cs.LGstat.ML
收藏

可证明 Transformer 利用多概念词语义实现高效的上下文学习

Provably Transformers Harness Multi-Concept Word Semantics for Efficient In-Context Learning

City University of Hong Kong(香港城市大学);RIKEN(日本理化学研究所);Nanyang Technological University(南洋理工大学);University of Tokyo(东京大学)

Dake Bu, Wei Huang, Andi Han, Atsushi Nitanda, Taiji Suzuki, Qingfu Zhang, Hau-San Wong

AI总结:

该研究通过细粒度数学分析,证明 Transformer 利用多概念词语义实现高效的上下文学习及分布外能力,并基于稀疏编码提示模型证明了指数级损失收敛。

摘要:

基于 Transformer 的大型语言模型(LLMs)展现出了非凡的创造力和涌现能力。现有的实证研究揭示了这些 LLMs 令人瞩目的涌现能力与其上下文学习(ICL)能力之间存在紧密联系,使其能够仅使用特定任务的提示来解决新任务,而无需进一步微调。另一方面,现有的实证和理论研究也表明,基于 Transformer 的 LLMs 背后,其多概念编码的语义表示存在线性规律性。然而,现有的理论工作未能理解这种规律性与 ICL 创新能力之间的联系。此外,先前的工作通常关注简化的、不切实际的场景,涉及线性 Transformer 或不切实际的损失函数,并且它们仅实现了线性或次线性的收敛速度。相比之下,这项工作提供了细粒度的数学分析,展示了 Transformer 如何利用单词的多概念语义来实现强大的 ICL 和出色的分布外 ICL 能力,从而深入理解 Transformer 如何为某些编码了多个跨概念语义的未见任务创新解决方案。受 LLMs 线性潜在几何结构的实证研究启发,该分析基于概念的低噪声稀疏编码提示模型。利用先进技术,这项工作展示了在高度非凸训练动态上的指数级 0-1 损失收敛,开创性地结合了 softmax 自注意力、ReLU 激活的 MLPs 和交叉熵损失的挑战。实证模拟证实了这些理论发现。

英文摘要:

Transformer-based large language models (LLMs) have displayed remarkable creative prowess and emergence capabilities. Existing empirical studies have revealed a strong connection between these LLMs' impressive emergence abilities and their in-context learning (ICL) capacity, allowing them to solve new tasks using only task-specific prompts without further fine-tuning. On the other hand, existing empirical and theoretical studies also show that there is a linear regularity of the multi-concept encoded semantic representation behind transformer-based LLMs. However, existing theoretical work fail to build up an understanding of the connection between this regularity and the innovative power of ICL. Additionally, prior work often focuses on simplified, unrealistic scenarios involving linear transformers or unrealistic loss functions, and they achieve only linear or sub-linear convergence rates. In contrast, this work provides a fine-grained mathematical analysis to show how transformers leverage the multi-concept semantics of words to enable powerful ICL and excellent out-of-distribution ICL abilities, offering insights into how transformers innovate solutions for certain unseen tasks encoded with multiple cross-concept semantics. Inspired by empirical studies on the linear latent geometry of LLMs, the analysis is based on a concept-based low-noise sparse coding prompt model. Leveraging advanced techniques, this work showcases the exponential 0-1 loss convergence over the highly non-convex training dynamics, which pioneeringly incorporates the challenges of softmax self-attention, ReLU-activated MLPs, and cross-entropy loss. Empirical simulations corroborate the theoretical findings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑