首页/论文/2007.02876

arXiv 2007.028762020-07-20stat.MLcs.LG

A Mathematical Theory of Attention

Mila（Mila（魁北克人工智能研究所））；Université de Montréal（蒙特利尔大学）；Microsoft Research Montreal（微软研究院蒙特利尔分院）；Microsoft（微软）

James Vuckovic, Aristide Baratin, Remi Tachet des Combes

英文摘要：

Attention is a powerful component of modern neural networks across a wide variety of domains. However, despite its ubiquity in machine learning, there is a gap in our understanding of attention from a theoretical point of view. We propose a framework to fill this gap by building a mathematically equivalent model of attention using measure theory. With this model, we are able to interpret self-attention as a system of self-interacting particles, we shed light on self-attention from a maximum entropy perspective, and we show that attention is actually Lipschitz-continuous (with an appropriate metric) under suitable assumptions. We then apply these insights to the problem of mis-specified input data; infinitely-deep, weight-sharing self-attention networks; and more general Lipschitz estimates for a specific type of attention studied in concurrent work.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

The Lipschitz Constant of Self-Attention

DeepMind（深度思维）

Lipschitz constants of neural networks have been explored in various contexts in deep learning, such as provable adversarial robustness, estimating Wasserstein distance, stabilising training of GANs, and formulating invertible neural networks. Such works have focused on bounding the Lipschitz constant of fully connected or convolutional networks, composed of linear maps and pointwise non-linearities. In this paper, we investigate the Lipschitz constant of self-attention, a non-linear neural network module widely used in sequence modelling. We prove that the standard dot-product self-attention …

2021-06-09cs.LGstat.ML

Lipschitz Normalization for Self-Attention Layers with Application to Graph Neural Networks

École Polytechnique（巴黎综合理工学院）；Huawei Technologies France（华为技术法国公司）

Attention based neural networks are state of the art in a large range of applications. However, their performance tends to degrade when the number of layers increases. In this work, we show that enforcing Lipschitz continuity by normalizing the attention scores can significantly improve the performance of deep attention models. First, we show that, for deep graph attention networks (GAT), gradient explosion appears during training, leading to poor performance of gradient-based training algorithms. To address this issue, we derive a theoretical analysis of the Lipschitz continuity of attention …

2021-09-13cs.LGstat.ML

On the Regularity of Attention

Microsoft（微软）；Mila（米拉研究所）；Université de Montréal（蒙特利尔大学）；Microsoft Research Montréal（微软蒙特利尔研究院）

Attention is a powerful component of modern neural networks across a wide variety of domains. In this paper, we seek to quantify the regularity (i.e. the amount of smoothness) of the attention operation. To accomplish this goal, we propose a new mathematical framework that uses measure theory and integral operators to model attention. We show that this framework is consistent with the usual definition, and that it captures the essential properties of attention. Then we use this framework to prove that, on compact domains, the attention operation is Lipschitz continuous and provide an estimate …

2021-02-10cs.LGstat.ML

Pay attention to your loss: understanding misconceptions about 1-Lipschitz neural networks

IRIT, Université Paul-Sabatier（保罗·萨巴蒂埃大学图卢兹信息学研究所）；IRT Saint-Exupéry（圣埃克苏佩里技术研究所）；IMT, Université Paul-Sabatier（保罗·萨巴蒂埃大学图卢兹数学研究所）

Lipschitz constrained networks have gathered considerable attention in the deep learning community, with usages ranging from Wasserstein distance estimation to the training of certifiably robust classifiers. However they remain commonly considered as less accurate, and their properties in learning are still not fully understood. In this paper we clarify the matter: when it comes to classification 1-Lipschitz neural networks enjoy several advantages over their unconstrained counterpart. First, we show that these networks are as accurate as classical ones, and can fit arbitrarily difficult bound…

2022-10-17cs.AIcs.LGstat.ML

CLIP: Cheap Lipschitz Training of Neural Networks

Friedrich-Alexander University Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）

Despite the large success of deep neural networks (DNN) in recent years, most neural networks still lack mathematical guarantees in terms of stability. For instance, DNNs are vulnerable to small or even imperceptible input perturbations, so called adversarial examples, that can cause false predictions. This instability can have severe consequences in applications which influence the health and safety of humans, e.g., biomedical imaging or autonomous driving. While bounding the Lipschitz constant of a neural network improves stability, most methods rely on restricting the Lipschitz constants of…

2022-10-31cs.LGmath.OCstat.ML

Analytical bounds on the local Lipschitz constants of affine-ReLU functions

University of Washington（华盛顿大学）

In this paper, we determine analytical bounds on the local Lipschitz constants of of affine functions composed with rectified linear units (ReLUs). Affine-ReLU functions represent a widely used layer in deep neural networks, due to the fact that convolution, fully-connected, and normalization functions are all affine, and are often followed by a ReLU activation function. Using an analytical approach, we mathematically determine upper bounds on the local Lipschitz constant of an affine-ReLU function, show how these bounds can be combined to determine a bound on an entire network, and discuss ho…

2020-08-14cs.LGstat.ML

更多相关论文收起更多论文

正交性：统一、高效的正交和1-利普希茨构建块库

Orthogonium : A Unified, Efficient Library of Orthogonal and 1-Lipschitz Building Blocks

Natural Intelligence Toulouse Institute, France（法国图卢兹自然智能研究所）；INRIA, Bordeaux, France（法国波尔多国家信息与自动化研究所）

Orthogonium 是一个统一高效的PyTorch库，提供正交和1-利普希茨层，以提升深度学习模型的鲁棒性和稳定性。

2026-01-20cs.LGstat.ML

AutoShuffleNet: Learning Permutation Matrices via an Exact Lipschitz Continuous Penalty in Deep Convolutional Neural Networks

Qualcomm AI Research（高通人工智能研究院）；UC Irvine（加州大学欧文分校）

ShuffleNet is a state-of-the-art light weight convolutional neural network architecture. Its basic operations include group, channel-wise convolution and channel shuffling. However, channel shuffling is manually designed empirically. Mathematically, shuffling is a multiplication by a permutation matrix. In this paper, we propose to automate channel shuffling by learning permutation matrices in network training. We introduce an exact Lipschitz continuous non-convex penalty so that it can be incorporated in the stochastic gradient descent to approximate permutation at high precision. Exact permu…

2019-01-24cs.LGstat.ML

注意力遇上事后可解释性：数学视角

Attention Meets Post-hoc Interpretability: A Mathematical Perspective

Universit\'e C\ ote d'Azur, Inria, CNRS

针对注意力权重能否作为模型解释的争论，本文从数学上分析简单注意力架构，比较事后解释与注意力解释，证明事后方法虽有局限，却能提供更有用的内部行为洞见。

2024-06-17cs.CLcs.LGstat.ML

A Mathematical Theory of Deep Convolutional Neural Networks for Feature Extraction

ETH Zurich（苏黎世联邦理工学院）

Deep convolutional neural networks have led to breakthrough results in numerous practical machine learning tasks such as classification of images in the ImageNet data set, control-policy-learning to play Atari games or the board game Go, and image captioning. Many of these applications first perform feature extraction and then feed the results thereof into a trainable classifier. The mathematical analysis of deep convolutional neural networks for feature extraction was initiated by Mallat, 2012. Specifically, Mallat considered so-called scattering networks based on a wavelet transform followed…

2017-10-24cs.AIcs.ITcs.LG

神经网络中由优化诱导的利普希茨连续性动力学

Optimization-Induced Dynamics of Lipschitz Continuity in Neural Networks

University of Galway（Galway大学）；Irish National Centre for Research Training in AI (CRT-AI)（爱尔兰人工智能研究培训中心）；Université Laval（拉瓦尔大学）；Canada-CIFAR AI Chair（加拿大CIFAR人工智能 chair）；Mila - Québec AI Institute（魁北克人工智能研究所）；University of Toronto（多伦多大学）；MBZUAI, UAE（阿联酋MBZUAI）

针对神经网络训练过程中利普希茨连续性演化规律研究不足的问题，提出基于SDE的SGD训练利普希茨连续性演化建模框架，识别三类核心驱动因素，实验验证理论与实际行为高度一致。

2025-11-14cs.AIcs.LGstat.ML

An Analysis of the Expressiveness of Deep Neural Network Architectures Based on Their Lipschitz Constants

University of Toronto（多伦多大学）；Vector Institute for Artificial Intelligence（向量人工智能研究所）

Deep neural networks (DNNs) have emerged as a popular mathematical tool for function approximation due to their capability of modelling highly nonlinear functions. Their applications range from image classification and natural language processing to learning-based control. Despite their empirical successes, there is still a lack of theoretical understanding of the representative power of such deep architectures. In this work, we provide a theoretical analysis of the expressiveness of fully-connected, feedforward DNNs with 1-Lipschitz activation functions. In particular, we characterize the exp…

2019-12-24cs.LGcs.ROcs.SY

Copy this Sentence

Carleton University（卡尔顿大学）

Attention is an operation that selects some largest element from some set, where the notion of largest is defined elsewhere. Applying this operation to sequence to sequence mapping results in significant improvements to the task at hand. In this paper we provide the mathematical definition of attention and examine its application to sequence to sequence models. We highlight the exact correspondences between machine learning implementations of attention and our mathematical definition. We provide clear evidence of effectiveness of attention mechanisms evaluating models with varying degrees of a…

2019-05-23cs.CLcs.LGstat.ML

A Mathematical Introduction to Generative Adversarial Nets (GAN)

Hong Kong University of Science and Technology（香港科技大学）

Generative Adversarial Nets (GAN) have received considerable attention since the 2014 groundbreaking work by Goodfellow et al. Such attention has led to an explosion in new ideas, techniques and applications of GANs. To better understand GANs we need to understand the mathematical foundation behind them. This paper attempts to provide an overview of GANs from a mathematical point of view. Many students in mathematics may find the papers on GANs more difficulty to fully understand because most of them are written from computer science and engineer point of view. The aim of this paper is to give…

2020-09-01cs.ITcs.LGmath.IT

Parsimonious Computing: A Minority Training Regime for Effective Prediction in Large Microarray Expression Data Sets

PES University（PES大学）；BITS Pilani K K Birla Goa Campus（比尔拉科学技术学院K K比拉果阿校区）；North Carolina State University（北卡罗来纳州立大学）；IIT Patna（印度理工学院巴特那分校）

Rigorous mathematical investigation of learning rates used in back-propagation in shallow neural networks has become a necessity. This is because experimental evidence needs to be endorsed by a theoretical background. Such theory may be helpful in reducing the volume of experimental effort to accomplish desired results. We leveraged the functional property of Mean Square Error, which is Lipschitz continuous to compute learning rate in shallow neural networks. We claim that our approach reduces tuning efforts, especially when a significant corpus of data has to be handled. We achieve remarkable…

2020-05-18cs.LGq-bio.QMstat.ML

通过雅可比矩阵和利普希茨常数理解深度学习中的优化

Understanding Optimization of Deep Learning via Jacobian Matrix and Lipschitz Constant

International Digital Economy Academy (IDEA)（国际数字经济学院（IDEA））

本文通过雅可比矩阵与利普希茨常数，系统分析深度学习优化中的梯度消失与爆炸问题，将优化方法分为显式和隐式两类，并实验验证理论，旨在提供全面理解而非提出新方法。

2023-11-12cs.CVcs.LGmath.OC