Transformer中上下文关系的表达能力
On the Expressive Power of Transformers for Contextual Relations
AI总结：
本文提出一种测度理论框架，将上下文关系建模为概率对象，揭示了softmax注意力与熵正则化最优传输的联系，并证明Transformer能近似任意上下文关系规则。
摘要：
Transformer架构在建模上下文关系方面取得了显著的实证成功，但对其表达能力的理解仍不清晰。本文引入一种测度理论框架，将上下文关系建模为概率对象，无论是条件分布还是联合分布（耦合）。这一视角揭示了标准softmax注意力与熵正则化最优传输之间的自然联系，为注意力提供了一种统一的视图，即作为底层亲和函数的归一化。在此框架内，我们利用标准softmax注意力和交替Sinkhorn归一化建立了上下文系统的通用近似定理。这些结果表明，Transformer架构能够近似任意上下文关系规则，且归一化的选择决定了这些关系的表示方式。此外，它们还提供了Transformers在建模上下文关系上有效的原因的原理性解释。
英文摘要：
Transformers have revolutionized machine learning by making attention a central mechanism for modeling interactions within a context. Despite the central role of attention, the theoretical capabilities of Transformers for representing contextual relations remain unclear. In this work, we address this question by developing a mathematical framework based on probability and optimal transport. We view a text as a distribution of its representations and attention as a probabilistic relation between them. This perspective reveals a connection between attention normalization and optimal transport: standard softmax normalization produces conditional relations, while Sinkhorn normalization produces joint relations with prescribed marginals. Thus, both mechanisms provide structured probabilistic relations from attention scores. Under mild conditions, we establish universal approximation results for both settings. We show that Transformer architectures with Sinkhorn normalization can approximate arbitrary contextual relations represented as joint probabilities, while standard softmax Transformers can approximate arbitrary contextual relations represented as conditional probabilities. These results provide a mathematical characterization of the expressive power of Transformers for contextual relations and show how the choice of normalization determines the probabilistic structure of the relations represented by attention.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
