实值Transformer的VC维与表达力
VC Dimension and Expressivity of Real-Valued Transformers
AI总结：
本文研究实值softmax注意力多层Transformer的VC维与表达力，给出上界$O(n^4)$/$O(n^6)$及下界$\Omega(n)$，并揭示其在对称函数表达和实数比特访问上的局限性。
AI中文摘要：
以往关于Transformer能力与局限性的结果以各种方式限制了Transformer的定义，而这里我们在极少额外假设下研究作用于实值的softmax注意力多层Transformer。应用实几何的结果，我们获得了此类Transformer的VC维和分裂VC维的上界（分别为$O(n^4)$和$O(n^6)$，其中$n$为输入长度）。反之，我们也构造了具体的Transformer来证明这些量的下界（每种情形均为$\Omega(n)$）。这些结果有一些显著推论。例如，在对称（置换不变）函数类中，我们证明Transformer可以一致地表达单符号字母表上的所有函数，非一致地表达双符号字母表上的所有函数，但无法（即使非一致地）表达六符号字母表上的某些函数。我们还证明了Transformer能访问实数中多少比特的局限性。
英文摘要：
Whereas previous results on abilities and limitations of transformers have restricted the definition of transformers in various ways, here we study softmax-attention, multi-layer transformers operating on real values, with very few additional assumptions. Applying results from real geometry, we obtain upper bounds on the VC dimension and split VC dimension of such transformers ($O(n^4)$ and $O(n^6)$, respectively, where $n$ is the input length). Conversely, we also construct specific transformers witnessing lower bounds on these quantities ($Ω(n)$ in each case). These results have some notable consequences. For example, within the class of symmetric (permutation-invariant) functions, we show that transformers can uniformly express all functions over an alphabet of one symbol and non-uniformly express all functions over an alphabet of two symbols, but cannot (even non-uniformly) express some functions over an alphabet of six symbols. We also prove limitations on how many bits of a real number a transformer can access.
