Training Implicit Networks for Image Deblurring using Jacobian-Free Backpropagation
Colorado School of Mines（科罗拉多矿业学院）；Brown University（布朗大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；University of California, Berkeley（加利福尼亚大学伯克利分校）
Recent efforts in applying implicit networks to solve inverse problems in imaging have achieved competitive or even superior results when compared to feedforward networks. These implicit networks only require constant memory during backpropagation, regardless of the number of layers. However, they are not necessarily easy to train. Gradient calculations are computationally expensive because they require backpropagating through a fixed point. In particular, this process requires solving a large linear system whose size is determined by the number of features in the fixed point iteration. This paper explores a recently proposed method, Jacobian-free Backpropagation (JFB), a backpropagation scheme that circumvents such calculation, in the context of image deblurring problems. Our results show that JFB is comparable against fine-tuned optimization schemes, state-of-the-art (SOTA) feedforward networks, and existing implicit networks at a reduced computational cost.
Colorado School of Mines（科罗拉多矿业学院）；Typal Research, Typal LLC；Alibaba Group (US), Damo Academy（阿里巴巴集团（美国），达摩院）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
A promising trend in deep learning replaces traditional feedforward networks with implicit networks. Unlike traditional networks, implicit networks solve a fixed point equation to compute inferences. Solving for the fixed point varies in complexity, depending on provided data and an error tolerance. Importantly, implicit networks may be trained with fixed memory costs in stark contrast to feedforward networks, whose memory requirements scale linearly with depth. However, there is no free lunch -- backpropagation through implicit networks often requires solving a costly Jacobian-based equation …
On the Convergence of Jacobian-Free Backpropagation for Optimal Control Problems with Implicit Hamiltonians
Department of Applied Mathematics and Statistics, Colorado School of Mines（应用数学与统计学系，科罗拉多矿业学院）；Department of Mathematical Sciences, Clemson University（数学科学系，克莱姆森大学）；Department of Mathematics, University of Tennessee Knoxville（数学系，田纳西大学肯尼思维尔分校）；Department of Mathematics, University of California Los Angeles（数学系，加州大学洛杉矶分校）
University of Denver（丹佛大学）；Lockheed Martin Space（洛克希德·马丁太空公司）
We describe, implement and test a novel method for training neural networks to estimate the Jacobian matrix $J$ of an unknown multivariate function $F$. The training set is constructed from finitely many pairs $(x,F(x))$ and it contains no explicit information about $J$. The loss function for backpropagation is based on linear approximations and on a nearest neighbor search in the sample data. We formally establish an upper bound on the uniform norm of the error, in operator norm, between the estimated Jacobian matrix provided by the algorithm and the actual Jacobian matrix, under natural assu…
In recent years, an increasing number of neural network models have included derivatives with respect to inputs in their loss functions, resulting in so-called double backpropagation for first-order optimization. However, so far no general description of the involved derivatives exists. Here, we cover a wide array of special cases in a very general Hilbert space framework, which allows us to provide optimized backpropagation rules for many real-world scenarios. This includes the reduction of calculations for Frobenius-norm-penalties on Jacobians by roughly a third for locally linear activation…
The local Lipschitz constant of a neural network is a useful metric with applications in robustness, generalization, and fairness evaluation. We provide novel analytic results relating the local Lipschitz constant of nonsmooth vector-valued functions to a maximization over the norm of the generalized Jacobian. We present a sufficient condition for which backpropagation always returns an element of the generalized Jacobian, and reframe the problem over this broad class of functions. We show strong inapproximability results for estimating Lipschitz constants of ReLU networks, and then formulate …
Physics-Informed Stochastic Configuration Machine: A Backpropagation-Free Neural Network with Fast Training for Nonlinear Differential Equations
School of Automation, Central South University（中南大学自动化学院）；Johns Hopkins University（约翰斯·霍普金斯大学）；China Institute of Water Resources and Hydropower Research（中国水利水电科学研究院）
Toulouse School of Economics, University of Toulouse Capitole（图卢兹第一大学图卢兹经济学院）；IRIT（图卢兹计算机科学研究所）；CNRS（法国国家科学研究中心）；Université de Toulouse（图卢兹大学）；Institut Universitaire de France (IUF)（法国大学研究院）；Université Côte d’Azur（蔚蓝海岸大学）；Laboratoire J. A. Dieudonné（J. A. 迪厄多内实验室）
Training convolutional neural networks (CNNs) with a strict Lipschitz constraint under the $l_{2}$ norm is useful for provable adversarial robustness, interpretable gradients and stable training. While $1$-Lipschitz CNNs can be designed by enforcing a $1$-Lipschitz constraint on each layer, training such networks requires each layer to have an orthogonal Jacobian matrix (for all inputs) to prevent the gradients from vanishing during backpropagation. A layer with this property is said to be Gradient Norm Preserving (GNP). In this work, we introduce a procedure to certify the robustness of $1$-L…
Training convolutional neural networks (CNNs) with a strict 1-Lipschitz constraint under the $l_{2}$ norm is useful for adversarial robustness, interpretable gradients and stable training. 1-Lipschitz CNNs are usually designed by enforcing each layer to have an orthogonal Jacobian matrix (for all inputs) to prevent the gradients from vanishing during backpropagation. However, their performance often significantly lags behind that of heuristic methods to enforce Lipschitz constraints where the resulting CNN is not \textit{provably} 1-Lipschitz. In this work, we reduce this gap by introducing (a…
In view of training increasingly complex learning architectures, we establish a nonsmooth implicit function theorem with an operational calculus. Our result applies to most practical problems (i.e., definable problems) provided that a nonsmooth form of the classical invertibility condition is fulfilled. This approach allows for formal subdifferentiation: for instance, replacing derivatives by Clarke Jacobians in the usual differentiation formulas is fully justified for a wide class of nonsmooth problems. Moreover this calculus is entirely compatible with algorithmic differentiation (e.g., back…
Seoul National University（首尔国立大学）；Hanyang University（汉阳大学）
Recently, several studies proposed methods to utilize some classes of optimization problems in designing deep neural networks to encode constraints that conventional layers cannot capture. However, these methods are still in their infancy and require special treatments, such as analyzing the KKT condition, for deriving the backpropagation formula. In this paper, we propose a new layer formulation called the fixed-point iteration (FPI) layer that facilitates the use of more complicated operations in deep networks. The backward FPI layer is also proposed for backpropagation, which is motivated b…
Variational methods are widely applied to ill-posed inverse problems for they have the ability to embed prior knowledge about the solution. However, the level of performance of these methods significantly depends on a set of parameters, which can be estimated through computationally expensive and time-consuming methods. In contrast, deep learning offers very generic and efficient architectures, at the expense of explainability, since it is often used as a black-box, without any fine control over its output. Deep unfolding provides a convenient approach to combine variational-based and deep lea…
KTH Royal Institute of Technology（瑞典皇家理工学院）；Columbia University（哥伦比亚大学）；Princeton University（普林斯顿大学）；Institute for Advanced Study（高等研究院）
The slow convergence rate and pathological curvature issues of first-order gradient methods for training deep neural networks, initiated an ongoing effort for developing faster $\mathit{second}$-$\mathit{order}$ optimization algorithms beyond SGD, without compromising the generalization error. Despite their remarkable convergence rate ($\mathit{independent}$ of the training batch size $n$), second-order algorithms incur a daunting slowdown in the $\mathit{cost}$ $\mathit{per}$ $\mathit{iteration}$ (inverting the Hessian matrix of the loss function), which renders them impractical. Very recentl…
University of Pisa（比萨大学）；University of Exeter（埃克塞特大学）
Since the recognition in the early nineties of the vanishing/exploding (V/E) gradient issue plaguing the training of neural networks (NNs), significant efforts have been exerted to overcome this obstacle. However, a clear solution to the V/E issue remained elusive so far. In this manuscript a new architecture of NN is proposed, designed to mathematically prevent the V/E issue to occur. The pursuit of approximate dynamical isometry, i.e. parameter configurations where the singular values of the input-output Jacobian are tightly distributed around 1, leads to the derivation of a NN's architectur…
