We consider in this work a system of two stochastic differential equations named the perturbed compositional gradient flow. By introducing a separation of fast and slow scales of the two equations, we show that the limit of the slow motion is given by an averaged ordinary differential equation. We then demonstrate that the deviation of the slow motion from the averaged equation, after proper rescaling, converges to a stochastic process with Gaussian inputs. This indicates that the slow motion can be approximated in the weak sense by a standard perturbed gradient flow or the continuous-time stochastic gradient descent algorithm that solves the optimization problem for a composition of two functions. As an application, the perturbed compositional gradient flow corresponds to the diffusion limit of the Stochastic Composite Gradient Descent (SCGD) algorithm for minimizing a composition of two expected-value functions in the optimization literatures. For the strongly convex case, such an analysis implies that the SCGD algorithm has the same convergence time asymptotic as the classical stochastic gradient descent algorithm. Thus it validates, at the level of continuous approximation, the effectiveness of using the SCGD algorithm in the strongly convex case.
University of California, Berkeley（加利福尼亚大学伯克利分校）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）
We investigate the connections between neural networks and simple building blocks in kernel space. In particular, using well established feature space tools such as direct sum, averaging, and moment lifting, we present an algebra for creating "compositional" kernels from bags of features. We show that these operations correspond to many of the building blocks of "neural tangent kernels (NTK)". Experimentally, we show that there is a correlation in test error between neural network architectures and the associated kernels. We construct a simple neural network architecture using only 3x3 convolu…
An averaging method is applied to derive effective approximation to the following singularly perturbed nonlinear stochastic damped wave equation νu_{tt}+u_t=\D u+f(u)+ν^α\dot{W} on an open bounded domain $D\subset\R^n$\,, $1\leq n\leq 3$\,. Here $ν>0$ is a small parameter characterising the singular perturbation, and $ν^α$\,, $0\leq α\leq 1/2$\,, parametrises the strength of the noise. Some scaling transformations and the martingale representation theorem yield the following effective approximation for small $ν$, u_t=\D u+f(u)+ν^α\dot{W} to an error of $\ord{ν^α}$\,.
In this article, we investigate averaging principle for stochastic hyperbolic-parabolic equations with two time-scales, in which both the slow and fast components are perturbed by multiplicative noises. Particularly, we prove that the rate of strong convergence for the slow component to the averaged dynamics is of order $1/2$, which significantly improves the order $1/4$ established in our previous work.
The convective Brinkman-Forchheimer equations describe the motion of incompressible fluid flows in a saturated porous medium. This work examines the multiscale stochastic convective Brinkman-Forchheimer (SCBF) equations perturbed by multiplicative Gaussian noise in two and three dimensional bounded domains. We establish a strong averaging principle for the stochastic 2D SCBF equations, which contains a fast time scale component governed by a stochastic reaction-diffusion equation with damping driven by multiplicative Gaussian noise. We exploit the Khasminkii's time discretization approach in t…
This article deals with the weak errors for averaging principle for a stochastic wave equation in a bounded interval $[0,L]$, perturbed by a oscillating term arising as the solution of a stochastic reaction-diffusion equation evolving with respect to the fast time. Under suitable conditions, it is proved that the rate of weak convergence to the averaged effective dynamics is of order $1$ via an asymptotic expansion approach.
This work is devoted to averaging principle of a two-time-scale stochastic partial differential equation on a bounded interval $[0, l]$, where both the fast and slow components are directly perturbed by additive noises. Under some regular conditions on drift coefficients, it is proved that the rate of weak convergence for the slow variable to the averaged dynamics is of order $1-\varepsilon$ for arbitrarily small $\varepsilon>0$. The proof is based on an asymptotic expansion of solutions to Kolmogorov equations associated with the multiple-time-scale system.
An ordinary differential equation perturbed by a null-recurrent diffusion will be considered in the case where the averaging type perturbation is strong only when a fast motion is close to the origin. The normal deviations of these solutions from the averaged motion are studied, and a central limit type theorem is proved. The limit process satisfies a linear equation driven by a Brownian motion time changed by the local time of the fast motion.
We propose Federated Accelerated Stochastic Gradient Descent (FedAc), a principled acceleration of Federated Averaging (FedAvg, also known as Local SGD) for distributed optimization. FedAc is the first provable acceleration of FedAvg that improves convergence speed and communication efficiency on various types of convex functions. For example, for strongly convex and smooth functions, when using $M$ workers, the previous state-of-the-art FedAvg analysis can achieve a linear speedup in $M$ if given $M$ rounds of synchronization, whereas FedAc only requires $M^{\frac{1}{3}}$ rounds. Moreover, we…
We study the small mass limit of the equation describing planar motion of a charged particle of a small mass $μ$ in a force field, containing a magnetic component, perturbed by a stochastic term. We regularize the problem by adding a small friction of intensity $\e>0$. We show that for all small but fixed frictions the small mass limit of $q_{μ, \e}$ gives the solution $q_\e$ to a stochastic first order equation, containing a noise-induced drift term. Then, by using a generalization of the classical averaging theorem for Hamiltonian systems by Freidlin and Wentzell, we take the limit of the sl…
The characteristic equation for a linear delay differential equation (DDE) has countably infinite roots on the complex plane. We deal with linear DDEs that are on the verge of instability, i.e. a pair of roots of the characteristic equation (eigenvalues) lie on the imaginary axis of the complex plane, and all other roots have negative real parts. We show that, when the system is perturbed by small noise, under an appropriate change of time scale, the law of the amplitude of projection onto the critical eigenspace is close to the law of a certain one-dimensional stochastic differential equation…
We consider deterministic and stochastic perturbations of dynamical systems with conservation laws in $\R^3$. The Landau-Lifshitz equation for the magnetization dynamics in ferromagnetics is a special case of our system. The averaging principle is a natural tool in such problems. But bifurcations in the set of invariant measures lead to essential modification in classical averaging. The limiting slow motion in this case, in general, is a stochastic process even if pure deterministic perturbations of a deterministic system are considered. The stochasticity is a result of instabilities in the no…
We consider a two-dimensional Hamiltonian system perturbed by a small diffusion term, whose coefficient is state-dependent and non-degenerate. As a result, the process consists of the fast motion along the level curves and slow motion across them. On finite time intervals, the large deviation principle applies, while on time scales that are inversely proportional to the size of the perturbation, the averaging principle holds, i.e., the projection of the process onto the Reeb graph converges to a Markov process. In our paper, we consider the intermediate time scales and prove the large deviatio…
We consider a class of stochastic kinetic equations, depending on two time scale separation parameters $ε$ and $δ$: the evolution equation contains singular terms with respect to $ε$, and is driven by a fast ergodic process which evolves at the time scale $t/δ^2$. We prove that when $(ε,δ)\to (0,0)$ the density converges to the solution of a linear diffusion PDE. This is a mixture of diffusion approximation in the PDE sense (with respect to the parameter $ε$) and of averaging in the probabilistic sense (with respect to the parameter $δ$). The proof employs stopping times arguments and a suitab…
This article concerns the large deviations regime and the consequent solution of the Kramers problem for a two-time scale stochastic system driven by a common jump noise signal perturbed in small intensity $\varepsilon>0$ and with accelerated jumps by intensity $\frac{1}{\varepsilon}$. We establish Freidlin-Wentzell estimates for the slow process of the multiscale system in the small noise limit $\varepsilon \rightarrow 0$ using the weak convergence approach to large deviations theory. The core of our proof is the reduction of the large deviations principle to the establishment of a stochastic…
