发表机构
California Institute of Technology; Lawrence Berkeley National Laboratory; University of California, Berkeley; ICSI（加州理工学院; 劳伦斯伯克利国家实验室; 加州大学伯克利分校; 国际计算机科学研究所）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文针对重尾测度上的算子学习，提出用增长更慢的注意力核替代softmax以避免集成坍缩，并通过两个基准验证了该方法的有效性。
AI 中文摘要
在概率测度上的算子学习可以通过变换器（transformer）实现。对于具有多项式尾部的测度，softmax中的指数加权可能导致相应的测度级注意力积分发散。这促使我们用增长更慢的函数替换指数函数。我们构建了两个用于测度上算子学习的基准，这些测度具有闭式目标。我们利用这些基准研究后归一化变换器中注意力核的增长和数据变换。在没有数据变换的情况下，softmax模型在两个重尾基准上都表现出集成坍缩，而三个增长更慢的核避免了坍缩。Symlog预处理使得softmax在矩阵求逆任务上避免坍缩，但在剪切交换任务上则不然。在高斯对照中，所有四个核表现相似。我们还考察了样本量如何影响经验能量和Wasserstein距离对尾部差异的敏感性。这些结果支持将增长更慢的注意力核作为后归一化变换器从重尾集成中学习的有效设计选择。
英文摘要
Operator learning on probability measures can be accomplished with transformers. For measures with polynomial tails, the exponential weighting in softmax can make the corresponding measure-level attention integrals diverge. This motivates replacing the exponential with slower-growing functions. We construct two benchmarks for operator learning on measures with closed-form targets. We use these benchmarks to study attention kernel growth and data transformation in post-norm transformers. Without data transformation, the softmax models exhibit ensemble collapse on both heavy-tailed benchmarks, while the three slower-growing kernels avoid collapse. Symlog preprocessing allows softmax to avoid collapse on the matrix inverse task but not on the sheared swap task. On the Gaussian control, all four kernels perform similarly. We also examine how sample size affects the sensitivity of empirical energy and Wasserstein distances to tail differences. These results support slower-growing attention kernels as an effective design choice for post-norm transformers learning from heavy-tailed ensembles.
Comments32 pages, 17 figures, accepted to NeurIPS 2026 Workshop on AI for Stochastic Dynamics
