arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.01367cs.CLstat.ML
收藏

Transformer 是通用的上下文学习器

Transformers are Universal In-context Learners

Shimane Univ.(岛根大学);Rice Univ.(莱斯大学);CNRS(法国国家科学研究中心);ENS(巴黎高等师范学院);PSL Univ.(巴黎文理研究大学)

Takashi Furuya, Maarten V. de Hoop, Gabriel Peyré

AI总结:

本文从 Wasserstein 连续性角度证明深层 Transformer 是通用上下文学习器,可在固定嵌入维度和头数下以任意精度逼近连续上下文映射,并能处理任意甚至无限数量的 token。

摘要:

Transformer 是定义“上下文映射”的深层架构,这种映射能够基于给定的一组 token(例如 NLP 应用中的提示或视觉 Transformer 中的一组图像块)来预测新的 token。本文特别研究这些架构处理任意大量上下文 token 的能力。为了在数学上统一刻画其表达能力,我们考虑映射以 token 的概率分布所表示的上下文为条件的情形,当 token 数量有限时该分布退化为离散分布。相应的光滑性概念对应于这些上下文之间 Wasserstein 距离意义下的连续性。我们证明深层 Transformer 具有通用性,能够在紧致 token 域上一致地以任意精度逼近连续的上下文映射。与现有结果相比,我们结果的一个关键之处在于:对于固定精度,单个 Transformer 可以处理任意(甚至无限)数量的 token。此外,它以固定的 token 嵌入维度运行(该维度不随精度提高而增大),并且头数固定(与维度成比例)。多头注意力层之间 MLP 的使用也受到明确控制。我们同时考虑非掩码注意力(如视觉 Transformer 所用)和掩码因果注意力(如 NLP 和时间序列应用所用)。我们借助时空提升来处理因果情形,将因果注意力分析为 token 概率分布上的映射。

英文摘要:

Transformers are deep architectures that define "in-context mappings" which enable predicting new tokens based on a given set of tokens (such as a prompt in NLP applications or a set of patches for a vision transformer). In this work, we study in particular the ability of these architectures to handle an arbitrarily large number of context tokens. To mathematically, uniformly address their expressivity, we consider the case that the mappings are conditioned on a context represented by a probability distribution of tokens which becomes discrete for a finite number of these. The relevant notion of smoothness then corresponds to continuity in terms of the Wasserstein distance between these contexts. We demonstrate that deep transformers are universal and can approximate continuous in-context mappings to arbitrary precision, uniformly over compact token domains. A key aspect of our results, compared to existing findings, is that for a fixed precision, a single transformer can operate on an arbitrary (even infinite) number of tokens. Additionally, it operates with a fixed embedding dimension of tokens (this dimension does not increase with precision) and a fixed number of heads (proportional to the dimension). The use of MLPs between multi-head attention layers is also explicitly controlled. We consider both unmasked attentions (as used for the vision transformer) and masked causal attentions (as used for NLP and time series applications). We tackle the causal setting leveraging a space-time lifting to analyze causal attention as a mapping over probability distributions of tokens.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑