arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.15503cs.LGstat.ML
收藏

Lipschitz连续变换的近似理论

Approximation Theory for Lipschitz Continuous Transformers

Faculty of Life and Medical Sciences, Department of Biomedical Engineering, Doshisha University, RIKEN AIP(生命与医学科学学院,生物医学工程系,大原大学,RIKEN AIP);Department of Applied Mathematics and Theoretical Physics, University of Cambridge(应用数学与理论物理系,剑桥大学)

Takashi Furuya, Davide Murari, Carola-Bibiane Schönlieb

AI总结:

本文提出了一类梯度下降型的上下文变换器,通过构造保证Lipschitz连续性,并在Lipschitz约束函数空间内证明了通用近似定理,为设计鲁棒的变换器架构提供了理论支持。

摘要:

稳定性与鲁棒性对于在安全敏感环境下部署变换器至关重要。一种有原则的方法是通过约束模型的Lipschitz常数来强制此类行为。然而,对于显式保持Lipschitz连续性的架构,尚缺乏近似理论保证。在本工作中,我们通过引入一类梯度下降型的上下文变换器,填补了这一空白。这些变换器通过构造确保Lipschitz连续性。我们实现了MLP和注意力块作为负梯度流的显式欧拉步骤,从而在不牺牲表达能力的情况下确保内在稳定性。我们证明了该类在Lipschitz约束函数空间内的通用近似定理。关键在于我们的分析采用测度论形式化,将变换器解释为在概率测度上的算子,从而获得与标记数量无关的近似保证。这些结果为设计鲁棒、Lipschitz连续的变换器架构提供了严谨的理论基础。

英文摘要:

Stability and robustness are critical for deploying Transformers in safety-sensitive settings. A principled way to enforce such behavior is to constrain the model's Lipschitz constant. However, approximation-theoretic guarantees for architectures that explicitly preserve Lipschitz continuity have yet to be established. In this work, we bridge this gap by introducing a class of gradient-descent-type in-context Transformers that are Lipschitz-continuous by construction. We realize both MLP and attention blocks as explicit Euler steps of negative gradient flows, ensuring inherent stability without sacrificing expressivity. We prove a universal approximation theorem for this class within a Lipschitz-constrained function space. Crucially, our analysis adopts a measure-theoretic formalism, interpreting Transformers as operators on probability measures, to yield approximation guarantees independent of token count. These results provide a rigorous theoretical foundation for the design of robust, Lipschitz continuous Transformer architectures.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑