As deep neural networks (DNNs) become deeper, the training time increases. In this perspective, multi-GPU parallel computing has become a key tool in accelerating the training of DNNs. In this paper, we introduce a novel methodology to construct a parallel neural network that can utilize multiple GPUs simultaneously from a given DNN. We observe that layers of DNN can be interpreted as the time step of a time-dependent problem and can be parallelized by emulating a parallel-in-time algorithm called parareal. The parareal algorithm consists of fine structures which can be implemented in parallel and a coarse structure which gives suitable approximations to the fine structures. By emulating it, the layers of DNN are torn to form a parallel structure which is connected using a suitable coarse network. We report accelerated and accuracy-preserved results of the proposed methodology applied to VGG-16 and ResNet-1001 on several datasets.
In this work, the Parareal algorithm is applied to evolution problems that admit good low-rank approximations and for which the dynamical low-rank approximation (DLRA) can be used as time stepper. Many discrete integrators for DLRA have recently been proposed, based on splitting the projected vector field or by applying projected Runge--Kutta methods. The cost and accuracy of these methods are mostly governed by the rank chosen for the approximation. These properties are used in a new method, called low-rank Parareal, in order to obtain a time-parallel DLRA solver for evolution problems. The a…
Parareal is a widely studied parallel-in-time method that can achieve meaningful speedup on certain problems. However, it is well known that the method typically performs poorly on non-diffusive equations. This paper analyzes linear stability and convergence for IMEX Runge-Kutta Parareal methods on non-diffusive equations. By combining standard linear stability analysis with a simple convergence analysis, we find that certain Parareal configurations can achieve parallel speedup on non-diffusive equations. These stable configurations all posses low iteration counts, large block sizes, and a lar…
Parareal is a well-studied algorithm for numerically integrating systems of time-dependent differential equations by parallelising the temporal domain. Given approximate initial values at each temporal sub-interval, the algorithm locates a solution in a fixed number of iterations using a predictor-corrector, stopping once a tolerance is met. This iterative process combines solutions located by inexpensive (coarse resolution) and expensive (fine resolution) numerical integrators. In this paper, we introduce a stochastic parareal algorithm aimed at accelerating the convergence of the determinist…
The time parallel solution of optimality systems arising in PDE constraint optimization could be achieved by simply applying any time parallel algorithm, such as Parareal, to solve the forward and backward evolution problems arising in the optimization loop. We propose here a different strategy by devising directly a new time parallel algorithm, which we call ParaOpt, for the coupled forward and backward non-linear partial differential equations. ParaOpt is inspired by the Parareal algorithm for evolution equations, and thus is automatically a two-level method. We provide a detailed convergenc…
In order to make the numerical simulation of atherosclerotic plaque growth feasible, a temporal homogenization approach is employed. The resulting macro-scale problem for the plaque growth can be further accelerated by using parallel time integration schemes, such as the parareal algorithm. However, the parallel scalability is dominated by the computational cost of the coarse propagator. Therefore, in this paper, an interpolation-based coarse propagator, which uses growth values from previously computed micro-scale problems, is introduced. For a simple model problem, it is shown that this appr…
Parareal in time intermediate targets methods for optimal control problem
UPMC Univ Paris 06, UMR 7598, Laboratoire Jacques-Louis Lions（巴黎第六大学）；Division of Applied Mathematics, Brown University（布朗大学应用数学系）；CEREMADE, Université Paris-Dauphine（巴黎第九大学）
Parareal methods for highly oscillatory dynamical systems
Bar-Ilan University（巴伊兰大学）；Georgia Institute of Technology（佐治亚理工学院）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；KTH Royal Institute of Technology（瑞典皇家理工学院）
We construct a space-time parallel method for solving parabolic partial differential equations by coupling the Parareal algorithm in time with overlapping domain decomposition in space. The goal is to obtain a discretization consisting of "local" problems that can be solved on parallel computers efficiently. However, this introduces significant sources of error that must be evaluated. Reformulating the original Parareal algorithm as a variational method and implementing a finite element discretization in space enables an adjoint-based a posteriori error analysis to be performed. Through an app…
In this work the parallel-in-time algorithm Parareal was applied to the ocean-circulation and sea-ice model FESOM2 developed by the Alfred-Wegener Institut (AWI). The climate model provides one time integration method and hence, the coarse and fine propagators were defined by time step width. The coarse method was executed at the CFL condition limit, while fine step-sizes where gradually refined. As a first assessment of the performance of Parareal a low-resolution test mesh was used with default settings provided by the AWI. An introduction to FESOM2 and the straightforward implementation of …
Stochastic parareal (SParareal) is a probabilistic variant of the popular parallel-in-time algorithm known as parareal. Similarly to parareal, it combines fine- and coarse-grained solutions to an ordinary differential equation (ODE) using a predictor-corrector (PC) scheme. The key difference is that carefully chosen random perturbations are added to the PC to try to accelerate the location of a stochastic solution to the ODE. In this paper, we derive superlinear and linear mean-square error bounds for SParareal applied to nonlinear systems of ODEs using different types of perturbations. We ill…
Parareal for diffusion problems with space- and time-dependent coefficients
Institute of Computational Science, Università della Svizzera italiana（瑞士意大利大学计算科学研究所）；Jülich Supercomputing Centre, Forschungszentrum Jülich GmbH（于利希超级计算中心，于利希研究中心有限公司）
Two of the most popular parallel-in-time methods are Parareal and multigrid-reduction-in-time (MGRIT). Recently, a general convergence theory was developed in Southworth (2019) for linear two-level MGRIT/Parareal that provides necessary and sufficient conditions for convergence, with tight bounds on worst-case convergence factors. This paper starts by providing a new and simplified analysis of linear error and residual propagation of Parareal, wherein the norm of error or residual propagation is given by one over the minimum singular value of a certain block bidiagonal operator. New discussion…
Asynchronous iterations are more and more investigated for both scaling and fault-resilience purpose on high performance computing platforms. While so far, they have been exclusively applied within space domain decomposition frameworks, this paper advocates a novel application direction targeting time-decomposed time-parallel approaches. Specifically, an asynchronous iterative model is derived from the Parareal scheme, for which convergence and speedup analysis are then conducted. It turned out that Parareal and async-Parareal feature very close convergence conditions, asymptotically equivalen…
Parallel-in-time methods, such as multigrid reduction-in-time (MGRIT) and Parareal, provide an attractive option for increasing concurrency when simulating time-dependent PDEs in modern high-performance computing environments. While these techniques have been very successful for parabolic equations, it has often been observed that their performance suffers dramatically when applied to advection-dominated problems or purely hyperbolic PDEs using standard rediscretization approaches on coarse grids. In this paper, we apply MGRIT or Parareal to the constant-coefficient linear advection equation, …
Convergence of Parareal for the Navier-Stokes equations depending on the Reynolds number
Università della Svizzera italiana（瑞士意大利语大学）；Institute of Computational Science, Università della Svizzera italiana（瑞士意大利语大学计算科学研究所）；Jülich Supercomputing Centre, Forschungszentrum Jülich（于利希超级计算中心，于利希研究中心）
This paper presents a new algorithm for the parallel in time (PiT) numerical simulation of time dependent partial/ordinary differential equations. We propose a reliable alternative to the well know parareal in time algorithm, by formulating the parallel in time problem algebraically and solve it using an adapted Bi-Conjugate gradient stabilized method. The proposed Parallel in time Stable Bi-Conjugate algorithm (PiTSBiCG) has a great potential in stabilizing the parallel resolution for a variety of problems. In this work, we describe the mathematical approach to the new algorithm and provide n…
Sequential numerical methods for integrating initial value problems (IVPs) can be prohibitively expensive when high numerical accuracy is required over the entire interval of integration. One remedy is to integrate in a parallel fashion, "predicting" the solution serially using a cheap (coarse) solver and "correcting" these values using an expensive (fine) solver that runs in parallel on a number of temporal subintervals. In this work, we propose a time-parallel algorithm (GParareal) that solves IVPs by modelling the correction term, i.e. the difference between fine and coarse solutions, using…
