Transformers are deep architectures that define "in-context mappings" which enable predicting new tokens based on a given set of tokens (such as a prompt in NLP applications or a set of patches for a vision transformer). In this work, we study in particular the ability of these architectures to handle an arbitrarily large number of context tokens. To mathematically, uniformly address their expressivity, we consider the case that the mappings are conditioned on a context represented by a probability distribution of tokens which becomes discrete for a finite number of these. The relevant notion of smoothness then corresponds to continuity in terms of the Wasserstein distance between these contexts. We demonstrate that deep transformers are universal and can approximate continuous in-context mappings to arbitrary precision, uniformly over compact token domains. A key aspect of our results, compared to existing findings, is that for a fixed precision, a single transformer can operate on an arbitrary (even infinite) number of tokens. Additionally, it operates with a fixed embedding dimension of tokens (this dimension does not increase with precision) and a fixed number of heads (proportional to the dimension). The use of MLPs between multi-head attention layers is also explicitly controlled. We consider both unmasked attentions (as used for the vision transformer) and masked causal attentions (as used for NLP and time series applications). We tackle the causal setting leveraging a space-time lifting to analyze causal attention as a mapping over probability distributions of tokens.
OmniNet: Omnidirectional Representations from Transformers
Google Research(谷歌研究院);Google Brain Team(谷歌大脑团队)
本文提出了基于Transformers的全向表示(OmniNet)。在OmniNet中,每个token都被允许参与整个网络中的所有token,而不是保持严格的水平接受域。这个过程也可以被解释为一种极端或密集的注意力机制,它具有整个网络宽度和深度的感受野。为此,全向注意力是通过元学习器学习的,元学习器本质上是另一种基于自我注意力的模型。为了减轻完全感受野注意力的计算成本,我们利用有效的自我注意力模型,如基于核的(Choromanski等人),低等级注意力(Wang et al.)和/或Big Bird(Zaheer等人)作为元学习者。在自回归语言建模(LM 1B,C4),机器翻译,远程竞技场(LRA)和图像识别上进行了广泛的实验。实验表明,OmniNet在这些任务上实现了相当大的改进,包括在LM 1B、WMT'14 En-De/En-Fr和Long Range Arena上实现了最先进的性能。此外,在视觉Transformer中使用全向表示导致在少样本学习和微调设置上的图像识别任务的显著改进。
Universal Approximation of Visual Autoregressive Transformers
The University of Chicago(芝加哥大学);University of New South Wales(新南威尔士大学);The University of Hong Kong(香港大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);The Simons Institute for the Theory of Computing(西蒙斯理论计算研究所)