To target challenges in differentiable optimization we analyze and propose strategies for derivatives of the Matérn kernel with respect to the smoothness parameter. This problem is of high interest in Gaussian processes modelling due to the lack of robust derivatives of the modified Bessel function of second kind with respect to order. In the current work we focus on newly identified series expansions for the modified Bessel function of second kind valid for complex orders. Using these expansions we obtain highly accurate results using the complex step method. Furthermore, we show that the evaluations using the recommended expansions are also more efficient than finite differences.
The accurate computation of the covariance matrix of fitted model parameters is a somewhat neglected task in Statistics. Algorithms are given for computing accurate covariance matrices derived from computing the Hessian matrix by numerical differentiation, and also for the covariance matrix of the posterior distribution of model parameters. Evaluations on two datasets where the Hessian could be computed analytically show that the numerical differentiation algorithm is very accurate.
In this article, we discuss two specific classes of models - Gaussian Mixture Copula models and Mixture of Factor Analyzers - and the advantages of doing inference with gradient descent using automatic differentiation. Gaussian mixture models are a popular class of clustering methods, that offers a principled statistical approach to clustering. However, the underlying assumption, that every mixing component is normally distributed, can often be too rigid for several real life datasets. In order to to relax the assumption about the normality of mixing components, a new class of parametric mixtu…
Banded matrices can be used as precision matrices in several models including linear state-space models, some Gaussian processes, and Gaussian Markov random fields. The aim of the paper is to make modern inference methods (such as variational inference or gradient-based sampling) available for Gaussian models with banded precision. We show that this can efficiently be achieved by equipping an automatic differentiation framework, such as TensorFlow or PyTorch, with some linear algebra operators dedicated to banded matrices. This paper studies the algorithmic aspects of the required operators, d…
Patterns of isolation-by-distance arise when population differentiation increases with increasing geographic distances. Patterns of isolation-by-distance are usually caused by local spatial dispersal, which explains why differences of allele frequencies between populations accumulate with distance. However, spatial variations of demographic parameters such as migration rate or population density can generate non-stationary patterns of isolation-by-distance where the rate at which genetic differentiation accumulates varies across space. To characterize non-stationary patterns of isolation-by-di…
Bayesian Optimization (BO) has shown great promise for the global optimization of functions that are expensive to evaluate, but despite many successes, standard approaches can struggle in high dimensions. To improve the performance of BO, prior work suggested incorporating gradient information into a Gaussian process surrogate of the objective, giving rise to kernel matrices of size $nd \times nd$ for $n$ observations in $d$ dimensions. Naïvely multiplying with (resp. inverting) these matrices requires $\mathcal{O}(n^2d^2)$ (resp. $\mathcal{O}(n^3d^3$)) operations, which becomes infeasible for…
In this paper we explore orthogonal systems in $\mathrm{L}_2(\mathbb{R})$ which give rise to a skew-Hermitian, tridiagonal differentiation matrix. Surprisingly, allowing the differentiation matrix to be complex leads to a particular family of rational orthogonal functions with favourable properties: they form an orthonormal basis for $\mathrm{L}_2(\mathbb{R})$, have a simple explicit formulae as rational functions, can be manipulated easily and the expansion coefficients are equal to classical Fourier coefficients of a modified function, hence can be calculated rapidly. We show that this famil…
This article provides an overview of some of the mathematical principles of Automatic Differentiation (AD). In particular, we summarise different descriptions of the Forward Mode of AD, like the matrix-vector product based approach, the idea of lifting functions to the algebra of dual numbers, the method of Taylor series expansion on dual numbers and the application of the push-forward operator, and explain why they all reduce to the same actual chain of computations. We further give a short mathematical description of some methods of higher-order Forward AD and, at the end of this paper, brie…
We introduce and develop a theory of orthogonality with respect to Sobolev inner products on the real line for sequences of functions with a tridiagonal, skew-Hermitian differentiation matrix. While a theory of such L2-orthogonal systems is well established, Sobolev orthogonality requires new concepts and their analysis. We characterise such systems completely as appropriately weighed Fourier transforms of orthogonal polynomials and present a number of illustrative examples, inclusive of a Sobolev-orthogonal system whose leading N coefficients can be computed in $\mathcal{O}(N \log N)$ operati…
Automatic differentiation (AD) is an ensemble of techniques that allow to evaluate accurate numerical derivatives of a mathematical function expressed in a computer programming language. In this paper we use AD for stating and solving solid mechanics problems. Given a finite element discretization of the domain, we evaluate the free energy of the solid as the integral of its strain energy density, and we make use of AD for directly obtaining the residual force vector and the tangent stiffness matrix of the problem, as the gradient and the Hessian of the free energy respectively. The result is …
The known Complex Step Derivative (CSD) method allows easy and accurate differentiation up to machine precision of real analytic functions by evaluating them a small imaginary step next to the real number line. The current paper proposes that derivatives of holomorphic functions can be calculated in a similar fashion by taking a small step in a quaternionic direction instead. It is demonstrated that in so doing the CSD properties of high accuracy and convergence are carried over to derivatives of holomorphic functions. To demonstrate the ease of implementation, numerical experiments were perfo…
Marchuk Institute of Numerical Mathematics of the Russian Academy of Sciences（俄罗斯科学院马尔丘克计算数学研究所）；HSE University（高等经济学院）；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术学院）
In scientific computing and machine learning applications, matrices and more general multidimensional arrays (tensors) can often be approximated with the help of low-rank decompositions. Since matrices and tensors of fixed rank form smooth Riemannian manifolds, one of the popular tools for finding low-rank approximations is to use Riemannian optimization. Nevertheless, efficient implementation of Riemannian gradients and Hessians, required in Riemannian optimization algorithms, can be a nontrivial task in practice. Moreover, in some cases, analytic formulas are not even available. In this pape…
BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所）；LVMT, ENPC, Institut Polytechnique de Paris, Univ Gustave Eiffel（法国国立路桥学校，巴黎理工学院，古斯塔夫·埃菲尔大学）；Machine Learning Group, Technical University of Berlin（柏林工业大学机器学习组）
Automatic differentiation (autodiff) has revolutionized machine learning. It allows to express complex computations by composing elementary ones in creative ways and removes the burden of computing their derivatives by hand. More recently, differentiation of optimization problem solutions has attracted widespread attention with applications such as optimization layers, and in bi-level problems such as hyper-parameter optimization and meta-learning. However, so far, implicit differentiation remained difficult to use for practitioners, as it often required case-by-case tedious mathematical deriv…
LVMT, ENPC, Institut Polytechnique de Paris, Univ Gustave Eiffel（巴黎高等路桥学院，巴黎理工学院，古斯塔夫·埃菲尔大学，LVMT实验室）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所（BIFOLD））；Machine Learning Group, Technical University of Berlin（柏林工业大学机器学习组）
A critical step in topology optimization (TO) is finding sensitivities. Manual derivation and implementation of the sensitivities can be quite laborious and error-prone, especially for non-trivial objectives, constraints and material models. An alternate approach is to utilize automatic differentiation (AD). While AD has been around for decades, and has also been applied in TO, wider adoption has largely been absent. In this educational paper, we aim to reintroduce AD for TO, and make it easily accessible through illustrative codes. In particular, we employ JAX, a high-performance Python libra…
