面向显式路径学习保证的高阶Transformer导数估计
Higher-Order Transformer Derivative Estimates for Explicit Pathwise Learning Guarantees
AI总结:
本文精确估计了现实Transformer模型所有阶的高阶导数,并基于指数遍历马尔可夫过程单条轨迹推导出显式路径泛化界,证明其学习速率为O(polylog(N)/√N)。
摘要:
在计算Transformer的完全显式泛化界时,一个固有挑战在于为给定的Transformer类$T$获得覆盖数估计。粗略估计依赖于对$T$中Transformer局部Lipschitz常数的统一上界,而更精细的估计则需要分析其高阶偏导数。遗憾的是,目前文献中尚不存在针对(现实)Transformer模型的这些精确高阶导数估计,因为由于Transformer块的复杂组合结构,它们在组合上非常微妙。本文通过精确估计Transformer模型所有阶的高阶导数来填补这一空白。我们考虑具有每块多个(非线性化)注意力头以及层归一化的现实Transformer。我们以注意力头数量、每个Transformer块的深度和宽度以及归一化层数量为参数,获得了所有常数的完全显式估计。此外,我们明确分析了各种标准激活函数选择(例如SWISH和GeLU)的影响。作为应用,我们为指数遍历马尔可夫过程的单条轨迹上的Transformer获得了显式路径泛化界,该界在固定的未来时间范围内有效。我们得出结论:现实世界中的Transformer能够以${O}(\operatorname{polylog}(N)/\sqrt{N})$的速率从单个马尔可夫过程轨迹的$N$个(非独立同分布)样本中学习。
英文摘要:
An inherent challenge in computing fully-explicit generalization bounds for transformers involves obtaining covering number estimates for the given transformer class $T$. Crude estimates rely on a uniform upper bound on the local-Lipschitz constants of transformers in $T$, and finer estimates require an analysis of their higher-order partial derivatives. Unfortunately, these precise higher-order derivative estimates for (realistic) transformer models are not currently available in the literature as they are combinatorially delicate due to the intricate compositional structure of transformer blocks. This paper fills this gap by precisely estimating all the higher-order derivatives of all orders for the transformer model. We consider realistic transformers with multiple (non-linearized) attention heads per block and layer normalization. We obtain fully-explicit estimates of all constants in terms of the number of attention heads, the depth and width of each transformer block, and the number of normalization layers. Further, we explicitly analyze the impact of various standard activation function choices (e.g. SWISH and GeLU). As an application, we obtain explicit pathwise generalization bounds for transformers on a single trajectory of an exponentially-ergodic Markov process valid at a fixed future time horizon. We conclude that real-world transformers can learn from $N$ (non-i.i.d.) samples of a single Markov process's trajectory at a rate of ${O}(\operatorname{polylog}(N)/\sqrt{N})$.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。