Approximation Theory for Lipschitz Continuous Transformers
Faculty of Life and Medical Sciences, Department of Biomedical Engineering, Doshisha University, RIKEN AIP(生命与医学科学学院,生物医学工程系,大原大学,RIKEN AIP);Department of Applied Mathematics and Theoretical Physics, University of Cambridge(应用数学与理论物理系,剑桥大学)
Stability and robustness are critical for deploying Transformers in safety-sensitive settings. A principled way to enforce such behavior is to constrain the model's Lipschitz constant. However, approximation-theoretic guarantees for architectures that explicitly preserve Lipschitz continuity have yet to be established. In this work, we bridge this gap by introducing a class of gradient-descent-type in-context Transformers that are Lipschitz-continuous by construction. We realize both MLP and attention blocks as explicit Euler steps of negative gradient flows, ensuring inherent stability without sacrificing expressivity. We prove a universal approximation theorem for this class within a Lipschitz-constrained function space. Crucially, our analysis adopts a measure-theoretic formalism, interpreting Transformers as operators on probability measures, to yield approximation guarantees independent of token count. These results provide a rigorous theoretical foundation for the design of robust, Lipschitz continuous Transformer architectures.
Mitigating Transformer Overconfidence via Lipschitz Regularization
University of Virginia(弗吉尼亚大学);Shenzhen Children’s Hospital(深圳市儿童医院);Purdue University(普渡大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Tencent(腾讯)
Hamilton-Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controls
Seoul National University(首尔国立大学);Institute of New Media and Communications(新媒体与传播研究所);Department of Electrical and Computer Engineering(电气与计算机工程系);Automation and Systems Research Institute(自动化与系统研究所)