首页/论文/2209.04293

arXiv 2209.042932022-09-09cs.LGcs.AImath.DGmath.MG

Robust-by-Design Classification via Unitary-Gradient Neural Networks

Scuola Superiore Sant’Anna（圣安娜高等研究院）

Fabio Brau, Giulio Rossolini, Alessandro Biondi, Giorgio Buttazzo

备注：: Under review

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

The use of neural networks in safety-critical systems requires safe and robust models, due to the existence of adversarial attacks. Knowing the minimal adversarial perturbation of any input x, or, equivalently, knowing the distance of x from the classification boundary, allows evaluating the classification robustness, providing certifiable predictions. Unfortunately, state-of-the-art techniques for computing such a distance are computationally expensive and hence not suited for online applications. This work proposes a novel family of classifiers, namely Signed Distance Classifiers (SDCs), that, from a theoretical perspective, directly output the exact distance of x from the classification boundary, rather than a probability score (e.g., SoftMax). SDCs represent a family of robust-by-design classifiers. To practically address the theoretical requirements of a SDC, a novel network architecture named Unitary-Gradient Neural Network is presented. Experimental results show that the proposed architecture approximates a signed distance classifier, hence allowing an online certifiable classification of x at the cost of a single inference.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

复值酉表示作为分类头用于改进深度神经网络中的不确定性量化

Complex-Valued Unitary Representations as Classification Heads for Improved Uncertainty Quantification in Deep Neural Networks

University of Tartu（塔尔图大学）；S Holding OÜ（3S控股公司）

本文提出一种复值酉表示作为分类头，通过改进深度神经网络的不确定性量化，显著提升了校准性能。

2026-02-17cs.AIcs.LG

Convolutional unitary or orthogonal recurrent neural networks

Rockefeller University（洛克菲勒大学）

Recurrent neural networks are extremely powerful yet hard to train. One of their issues is the vanishing gradient problem, whereby propagation of training signals may be exponentially attenuated, freezing training. Use of orthogonal or unitary matrices, whose powers neither explode nor decay, has been proposed to mitigate this issue, but their computational expense has hindered their use. Here we show that in the specific case of convolutional RNNs, we can define a convolutional exponential and that this operation transforms antisymmetric or anti-Hermitian convolution kernels into orthogonal o…

2023-02-14cond-mat.stat-mechcs.AIcs.LG

基于参数平移规则的酉光神经网络梯度计算

Gradients of unitary optical neural networks using parameter-shift rule

IEIT SYSTEMS Co., Ltd（IEIT SYSTEMS有限公司）；IEIT SYSTEMS (Beijing) Co., Ltd（IEIT SYSTEMS（北京）有限公司）

针对反向传播在光神经网络中受物理约束难以实现的问题，本文提出将参数平移规则适配于马赫-曾德尔干涉仪网格构建的酉光神经网络，利用光学干涉的傅里叶特性从硬件测量得到精确解析梯度，为光计算系统的硬件化训练提供新方案。

2025-06-13cs.ETcs.LGphysics.optics

Full-Capacity Unitary Recurrent Neural Networks

University of Washington（华盛顿大学）；Mitsubishi Electric Research Laboratories (MERL)（三菱电机研究实验室）

Recurrent neural networks are powerful models for processing sequential data, but they are generally plagued by vanishing and exploding gradient problems. Unitary recurrent neural networks (uRNNs), which use unitary recurrence matrices, have recently been proposed as a means to avoid these issues. However, in previous experiments, the recurrence matrices were restricted to be a product of parameterized unitary matrices, and an open question remains: when does such a parameterization fail to represent all unitary matrices, and how does this restricted representational capacity limit what can be…

2016-10-31cs.LGcs.NEstat.ML

projUNN: efficient method for training deep networks with unitary matrices

MIT（麻省理工学院）；Meta AI, FAIR（Meta人工智能研究院，基础人工智能研究）；NYU（纽约大学）；Turing Inc.（图灵公司）

In learning with recurrent or very deep feed-forward networks, employing unitary matrices in each layer can be very effective at maintaining long-range stability. However, restricting network parameters to be unitary typically comes at the cost of expensive parameterizations or increased training runtime. We propose instead an efficient method based on rank-$k$ updates -- or their rank-$k$ approximation -- that maintains performance at a nearly optimal training runtime. We introduce two variants of this method, named Direct (projUNN-D) and Tangent (projUNN-T) projected Unitary Neural Networks,…

2022-10-13cs.AIcs.LGquant-ph

Learning Unitaries by Gradient Descent

Massachusetts Institute of Technology（麻省理工学院）；University of Oxford（牛津大学）

We study the hardness of learning unitary transformations in $U(d)$ via gradient descent on time parameters of alternating operator sequences. We provide numerical evidence that, despite the non-convex nature of the loss landscape, gradient descent always converges to the target unitary when the sequence contains $d^2$ or more parameters. Rates of convergence indicate a "computational phase transition." With less than $d^2$ parameters, gradient descent converges to a sub-optimal solution, whereas with more than $d^2$ parameters, gradient descent converges exponentially to an optimal solution.

2020-02-18cs.LGmath-phmath.MP

更多相关论文收起更多论文

A Projection Algorithm for the Unitary Weights

University of California, Los Angeles（加利福尼亚大学洛杉矶分校）

Unitary neural networks are promising alternatives for solving the exploding and vanishing activation/gradient problem without the need for explicit normalization that reduces the inference speed. However, they often require longer training time due to the additional unitary constraints on their weight matrices. Here we show a novel algorithm using a backpropagation technique with Lie algebra for computing approximated unitary weights from their pre-trained, non-unitary counterparts. The unitary networks initialized with these approximations can reach the desired accuracies much faster, mitiga…

2021-02-19cs.AIcs.LG

Complex Unitary Recurrent Neural Networks using Scaled Cayley Transform

University of Kentucky（肯塔基大学）

Recurrent neural networks (RNNs) have been successfully used on a wide range of sequential data problems. A well known difficulty in using RNNs is the \textit{vanishing or exploding gradient} problem. Recently, there have been several different RNN architectures that try to mitigate this issue by maintaining an orthogonal or unitary recurrent weight matrix. One such architecture is the scaled Cayley orthogonal recurrent neural network (scoRNN) which parameterizes the orthogonal recurrent weight matrix through a scaled Cayley transform. This parametrization contains a diagonal scaling matrix co…

2019-02-25cs.LGstat.ML

Learning Unitary Operators with Help From u(n)

ETH Zurich（苏黎世联邦理工学院）；Weill Cornell Medical（威尔康奈尔医学院）

A major challenge in the training of recurrent neural networks is the so-called vanishing or exploding gradient problem. The use of a norm-preserving transition operator can address this issue, but parametrization is challenging. In this work we focus on unitary operators and describe a parametrization using the Lie algebra $\mathfrak{u}(n)$ associated with the Lie group $U(n)$ of $n \times n$ unitary matrices. The exponential map provides a correspondence between these spaces, and allows us to define a unitary matrix using $n^2$ real coefficients relative to a basis of the Lie algebra. The pa…

2017-01-10cs.LGstat.ML

GAM: Explainable Visual Similarity and Classification via Gradient Activation Maps

The Open University（开放大学）；Microsoft（微软）；Tel-Aviv University（特拉维夫大学）；Bar-Ilan University（巴伊兰大学）；Technion（以色列理工学院）；Tel Aviv University（特拉维夫大学）

We present Gradient Activation Maps (GAM) - a machinery for explaining predictions made by visual similarity and classification models. By gleaning localized gradient and activation information from multiple network layers, GAM offers improved visual explanations, when compared to existing alternatives. The algorithmic advantages of GAM are explained in detail, and validated empirically, where it is shown that GAM outperforms its alternatives across various tasks and datasets.

2021-09-02cs.AIcs.CVcs.LG

加速用于均质化问题的共轭梯度求解器的单位神经算子

Accelerating Conjugate Gradient Solvers for Homogenization Problems with Unitary Neural Operators

Institute of Applied Mechanics, Data Analytics in Engineering（应用力学研究所，工程数据化分析中心）；University of Stuttgart（斯图加特大学）

UNO-CG通过结合机器学习预条件器和经典求解器，加速均质化问题的求解，减少迭代次数并保持高性能。

2026-02-04cs.LGcs.NAmath.NA

Input-Output Equivalence of Unitary and Contractive RNNs

UCLA（加州大学洛杉矶分校）；NYU（纽约大学）

Unitary recurrent neural networks (URNNs) have been proposed as a method to overcome the vanishing and exploding gradient problem in modeling data with long-term dependencies. A basic question is how restrictive is the unitary constraint on the possible input-output mappings of such a network? This work shows that for any contractive RNN with ReLU activations, there is a URNN with at most twice the number of hidden states and the identical input-output mapping. Hence, with ReLU activations, URNNs are as expressive as general RNNs. In contrast, for certain smooth activations, it is shown that t…

2019-10-30cs.LGstat.ML

Stochastic Gradient Descent with Exponential Convergence Rates of Expected Classification Errors

The University of Tokyo（东京大学）；RIKEN（理化学研究所）

We consider stochastic gradient descent and its averaging variant for binary classification problems in a reproducing kernel Hilbert space. In the traditional analysis using a consistency property of loss functions, it is known that the expected classification error converges more slowly than the expected risk even when assuming a low-noise condition on the conditional label probabilities. Consequently, the resulting rate is sublinear. Therefore, it is important to consider whether much faster convergence of the expected classification error can be achieved. In recent research, an exponential …

2022-07-25cs.LGmath.OCstat.ML