We introduce Multi-Iteration Stochastic Optimizers, a novel class of first-order stochastic methods that control the relative $L^2$ error using successive control variates along the iteration path. By exploiting correlations between iterates, these control variates reduce the estimator's variance, making an accurate mean gradient estimation computationally affordable. Our approach centers on the Multi-Iteration stochastiC Estimator (MICE), which can be seamlessly coupled with any first-order stochastic optimizer due to its non-intrusive design. The algorithm adaptively selects which iterates to include in its index set. We provide both an error analysis of MICE and a convergence analysis for Multi-Iteration Stochastic Optimizers across various problem classes, including some non-convex cases. In the smooth, strongly convex setting, we demonstrate that to approximate a minimizer within a tolerance $tol$, SGD-MICE requires, on average, $O(tol^{-1})$ stochastic gradient evaluations, compared to $O(tol^{-1}\log(tol^{-1}))$ for SGD with adaptive batch sizes. In numerical experiments, SGD-MICE achieved the desired tolerance with fewer than 3\% of the gradient evaluations required by adaptive batch SGD. Additionally, MICE offers a straightforward stopping criterion based on the gradient norm, validated through consistency tests. To assess its efficiency, we present examples using both SGD-MICE and Adam-MICE, including a stochastic adaptation of the Rosenbrock function and logistic regression on various datasets. Compared to SGD, SAG, SAGA, SVRG, and SARAH, our approach consistently reduces the gradient sampling cost without the need for extensive parameter tuning.
In this paper, a proof of asymptotic stability for the combined system-optimizer dynamics associated with a class of real-time methods for equality constrained nonlinear model predictive control is presented. General Q-linearly convergent online optimization methods are considered and asymptotic stability results are derived for the case where a single iteration of the optimizer is carried out per sampling time. In particular, it is shown that, if the underlying sampling time is sufficiently short, asymptotic stability can be guaranteed. The results constitute an extension to existing attracti…
Product reuse and recovery is an efficient tool that helps companies to simultaneously address economic and environmental dimensions of sustainability. This paper presents a novel problem for stock management of reusable products in a single-vendor, multi-product, multi-retailer network. Several constraints, such as the maximum budget, storage capacity, number of orders, etc., are considered in their stochastic form to provide a more realistic framework. The presented problem is formulated as a constrained nonlinear mathematical model. The chance-constrained programming method is suggested to …
Practical results have shown that deep learning optimizers using small constant learning rates, hyperparameters close to one, and large batch sizes can find the model parameters of deep neural networks that minimize the loss functions. We first show theoretical evidence that the momentum method (Momentum) and adaptive moment estimation (Adam) perform well in the sense that the upper bound of the theoretical performance measure is small with a small constant learning rate, hyperparameters close to one, and a large batch size. Next, we show that there exists a batch size called the critical batc…
This paper studies a continuous-time stochastic linear-quadratic (SLQ) optimal control problem on infinite-horizon. A data-driven policy iteration algorithm is proposed to solve the SLQ problem. Without knowing three system coefficient matrices, this algorithm uses the collected data to iteratively approximate a solution of the corresponding stochastic algebraic Riccati equation (SARE). A simulation example is provided to illustrate the effectiveness and applicability of the algorithm.
When measuring the value of a function to be minimized is not only expensive but also with noise, the popular simultaneous perturbation stochastic approximation (SPSA) algorithm requires only two function values in each iteration. In this paper, we propose a method requiring only one function measurement value per iteration in the average sense. We prove the strong convergence and asymptotic normality of the new algorithm. Experimental results show the effectiveness and potential of our algorithm.
We study the convergence of stochastic fixed point iterations in the consistent case (in the sense of Butnariu and Flåm (1995)) in several different settings, under decreasingly restrictive regularity assumptions of the fixed point mappings. The iterations are Markov chains and, for the purposes of this study, convergence is understood in very restrictive terms. We show that sufficient conditions for geometric (linear) convergence in expectation of stochastic projection algorithms presented in Nedić (2011), are in fact necessary for geometric (linear) convergence in expectation more generally …
We present an optimization-based approach to stochastic control problems with nonclassical information structures. We cast these problems equivalently as optimization prob- lems on joint distributions. The resulting problems are necessarily nonconvex. Our approach to solving them is through convex relaxation. We solve the instance solved by Bansal and Basar with a particular application of this approach that uses the data processing inequality for constructing the convex relaxation. Using certain f-divergences, we obtain a new, larger set of inverse optimal cost functions for such problems. In…
We present a stochastic variance-reduced heavy ball power iteration algorithm for solving PCA and provide a convergence analysis for it. The algorithm is an extension of heavy ball power iteration, incorporating a step size so that progress can be controlled depending on the magnitude of the variance of stochastic gradients. The algorithm works with any size of the mini-batch, and if the step size is appropriately chosen, it attains global linear convergence to the first eigenvector of the covariance matrix in expectation. The global linear convergence result in expectation is analogous to tho…
Reference [11] investigated the almost sure weak convergence of block-coordinate fixed point algorithms and discussed their applications to nonlinear analysis and optimization. This algorithmic framework features random sweeping rules to select arbitrarily the blocks of variables that are activated over the course of the iterations and it allows for stochastic errors in the evaluation of the operators. The present paper establishes results on the mean-square and linear convergence of the iterates. Applications to monotone operator splitting and proximal optimization algorithms are presented.
We consider a stochastic optimal exit time feedback control problem. The Bellman equation is solved approximatively via the Policy Iteration algorithm on a polynomial ansatz space by a sequence of linear equations. As high degree multi-polynomials are needed, the corresponding equations suffer from the curse of dimensionality even in moderate dimensions. We employ tensor-train methods to account for this problem. The approximation process within the Policy Iteration is done via a Least-Squares ansatz and the integration is done via Monte-Carlo methods. Numerical evidences are given for the (mu…
This paper presents a novel value iteration (VI) algorithm for finding the optimal control for a kind of infinite-horizon stochastic linear quadratic (SLQ) problem with unknown systems. First, an off-line algorithm is estabilished to obtain the optimal feedback control of our problem. Then, based on the off-line algorithm, the VI-based model-free algorithm and its convergence proof is provided. The main feature of the model-free algorithm is that a stabilizing control is not needed to initiate the algorithm. Finally, we validate our results with a simulation example.
Recent results of Ye and Hansen, Miltersen and Zwick show that policy iteration for one or two player (perfect information) zero-sum stochastic games, restricted to instances with a fixed discount rate, is strongly polynomial. We show that policy iteration for mean-payoff zero-sum stochastic games is also strongly polynomial when restricted to instances with bounded first mean return time to a given state. The proof is based on methods of nonlinear Perron-Frobenius theory, allowing us to reduce the mean-payoff problem to a discounted problem with state dependent discount rate. Our analysis als…
This paper proves several Tauberian theorems for general iterations of operators, and provides two applications to zero-sum stochastic games where the total payoff is a weighted sum of the stage payoffs. The first application is to provide conditions under which the existence of the asymptotic value implies the convergence of the values of the weighted game, as players get more and more patient. The second application concerns stochastic games with finite state space and action sets. This paper builds a simple class of asymptotically optimal strategies in the weighted game, that at each stage …
This work proposes block-coordinate fixed point algorithms with applications to nonlinear analysis and optimization in Hilbert spaces. The asymptotic analysis relies on a notion of stochastic quasi-Fejér monotonicity, which is thoroughly investigated. The iterative methods under consideration feature random sweeping rules to select arbitrarily the blocks of variables that are activated over the course of the iterations and they allow for stochastic errors in the evaluation of the operators. Algorithms using quasinonexpansive operators or compositions of averaged nonexpansive operators are cons…
Distributed coordination algorithms (DCA) carry out information processing processes among a group of networked agents without centralized information fusion. Though it is well known that DCA characterized by an SIA (stochastic, indecomposable, aperiodic) matrix generate consensus asymptotically via synchronous iterations, the dynamics of DCA with asynchronous iterations have not been studied extensively, especially when viewed as stochastic processes. This paper aims to show that for any given irreducible stochastic matrix, even non-SIA, the corresponding DCA lead to consensus successfully vi…
We systematically review the Variational Optimization, Variational Inference and Stochastic Search perspectives on sampling-based dynamic optimization and discuss their connections to state-of-the-art optimizers and Stochastic Optimal Control (SOC) theory. A general convergence and sample complexity analysis on the three perspectives is provided through the unifying Stochastic Search perspective. We then extend these frameworks to their distributed versions for multi-agent control by combining them with consensus Alternating Direction Method of Multipliers (ADMM) to decouple the full problem i…
We consider stochastic control models with Borel spaces and universally measurable policies. For such models the standard policy iteration is known to have difficult measurability issues and cannot be carried out in general. We present a mixed value and policy iteration method that circumvents this difficulty. The method allows the use of stationary policies in computing the optimal cost function, in a manner that resembles policy iteration. It can also be used to address similar difficulties of policy iteration in the context of upper and lower semicontinuous models. We analyze the convergenc…
Recently, Sidford, Wang, Wu and Ye (2018) developed an algorithm combining variance reduction techniques with value iteration to solve discounted Markov decision processes. This algorithm has a sublinear complexity when the discount factor is fixed. Here, we extend this approach to mean-payoff problems, including both Markov decision processes and perfect information zero-sum stochastic games. We obtain sublinear complexity bounds, assuming there is a distinguished state which is accessible from all initial states and for all policies. Our method is based on a reduction from the mean payoff pr…
Kyungpook National University（庆北国立大学）；KINDI Centre for Computing Research, Qatar University（卡塔尔大学KINDI计算研究中心）；Nanyang Technological University（南洋理工大学）
Non-parametric tests can determine the better of two stochastic optimization algorithms when benchmarking results are ordinal, like the final fitness values of multiple trials. For many benchmarks, however, a trial can also terminate once it reaches a pre-specified target value. When only some trials reach the target value, two variables characterize a trial's outcome: the time it takes to reach the target value (or not) and its final fitness value. This paper describes a simple way to impose linear order on this two-variable trial data set so that traditional non-parametric methods can determ…
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Pontificia Universidad Católica de Chile（智利天主教大学）
We study stochastic monotone inclusion problems, which widely appear in machine learning applications, including robust regression and adversarial learning. We propose novel variants of stochastic Halpern iteration with recursive variance reduction. In the cocoercive -- and more generally Lipschitz-monotone -- setup, our algorithm attains $ε$ norm of the operator with $\mathcal{O}(\frac{1}{ε^3})$ stochastic operator evaluations, which significantly improves over state of the art $\mathcal{O}(\frac{1}{ε^4})$ stochastic operator evaluations required for existing monotone inclusion solvers applie…
Optimizer as Detector: Stochastic Gradient Descent for Latent Mixture Models
Institute of Operations Research and Analytics, National University of Singapore（新加坡国立大学运筹学与分析研究所）；NUS Business School, National University of Singapore（新加坡国立大学商学院）；School of Management, University of Science and Technology of China（中国科学技术大学管理学院）
University of Chicago（芝加哥大学）；Tencent AI Lab（腾讯人工智能实验室）
We present novel minibatch stochastic optimization methods for empirical risk minimization problems, the methods efficiently leverage variance reduced first-order and sub-sampled higher-order information to accelerate the convergence speed. For quadratic objectives, we prove improved iteration complexity over state-of-the-art under reasonable assumptions. We also provide empirical evidence of the advantages of our method compared to existing approaches in the literature.
The Chinese University of Hong Kong（香港中文大学）；Tencent AI Lab（腾讯人工智能实验室）；University of California, Davis（加州大学戴维斯分校）；University of Rochester（罗切斯特大学）
Regularized empirical risk minimization problem with linear predictor appears frequently in machine learning. In this paper, we propose a new stochastic primal-dual method to solve this class of problems. Different from existing methods, our proposed methods only require O(1) operations in each iteration. We also develop a variance-reduction variant of the algorithm that converges linearly. Numerical experiments suggest that our methods are faster than existing ones such as proximal SGD, SVRG and SAGA on high-dimensional problems.
University of Augsburg（奥格斯堡大学）；Tensor AI Solutions GmbH（Tensor AI解决方案有限公司）；Ulm University（乌尔姆大学）；Centre for Advanced Analytics and Predictive Sciences, University of Augsburg（奥格斯堡大学高级分析与预测科学中心）
University of Edinburgh（爱丁堡大学）；Toshiba Medical Visualization Systems（东芝医疗可视化系统公司）
In this paper we describe improvements to the particle swarm optimizer (PSO) made by inclusion of an unscented Kalman filter to guide particle motion. We demonstrate the effectiveness of the unscented Kalman filter PSO by comparing it with the original PSO algorithm and its variants designed to improve performance. The PSOs were tested firstly on a number of common synthetic benchmarking functions, and secondly applied to a practical three-dimensional image registration problem. The proposed methods displayed better performances for 4 out of 8 benchmark functions, and reduced the target regist…
Indian Institute of Information Technology, Allahabad（印度阿拉哈巴德信息技术学院）；PathPartner Technology Pvt. Ltd.（PathPartner科技私人有限公司）；Indian Statistical Institute（印度统计研究所）；Techno India University（Techno印度大学）
The convolutional neural networks (CNNs) are generally trained using stochastic gradient descent (SGD) based optimization techniques. The existing SGD optimizers generally suffer with the overshooting of the minimum and oscillation near minimum. In this paper, we propose a new approach, hereafter referred as AdaInject, for the gradient descent optimizers by injecting the second order moment into the first order moment. Specifically, the short-term change in parameter is used as a weight to inject the second order moment in the update rule. The AdaInject optimizer controls the parameter update,…
We consider zero-sum stochastic games with finite state and action spaces, perfect information, mean payoff criteria, without any irreducibility assumption on the Markov chains associated to strategies (multichain games). The value of such a game can be characterized by a system of nonlinear equations, involving the mean payoff vector and an auxiliary vector (relative value or bias). We develop here a policy iteration algorithm for zero-sum stochastic games with mean payoff, following an idea of two of the authors (Cochet-Terrasson and Gaubert, C. R. Math. Acad. Sci. Paris, 2006). The algorith…
In this paper, we consider the problem of finding an almost surely common fixed point of a family of paracontraction maps indexed on a probability space, which we refer to as the stochastic feasibility problem. We show that a random iteration of paracontraction maps driven by an ergodic stationary sequence converges, with probability one, to a solution of the stochastic feasibility problem, provided a solution exists. As applications, we obtain non-white noise randomized algorithms to solve the stochastic convex feasibility problem and the problem of finding an almost surely common zero of a c…
We study the convergence of random function iterations for finding an invariant measure of the corresponding Markov operator. We call the problem of finding such an invariant measure the stochastic fixed point problem. This generalizes earlier work studying the stochastic feasibility problem}, namely, to find points that are, with probability 1, fixed points of the random functions [Hermer, Luke, Sturm, 2019]. When no such points exist, the stochastic feasibility problem is called inconsistent, but still under certain assumptions, the more general stochastic fixed point problem has a solution …
University of California, Berkeley（加州大学伯克利分校）；Johns Hopkins University（约翰斯·霍普金斯大学）；Mila（米拉研究所）；Université de Montréal（蒙特利尔大学）；McGill University（麦吉尔大学）；Microsoft Research（微软研究院）
We study the stochastic bilinear minimax optimization problem, presenting an analysis of the same-sample Stochastic ExtraGradient (SEG) method with constant step size, and presenting variations of the method that yield favorable convergence. In sharp contrasts with the basic SEG method whose last iterate only contracts to a fixed neighborhood of the Nash equilibrium, SEG augmented with iteration averaging provably converges to the Nash equilibrium under the same standard settings, and such a rate is further improved by incorporating a scheduled restarting procedure. In the interpolation settin…
A central issue in machine learning is how to train models on sensitive user data. Industry has widely adopted a simple algorithm: Stochastic Gradient Descent with noise (a.k.a. Stochastic Gradient Langevin Dynamics). However, foundational theoretical questions about this algorithm's privacy loss remain open -- even in the seemingly simple setting of smooth convex losses over a bounded domain. Our main result resolves these questions: for a large range of parameters, we characterize the differential privacy up to a constant factor. This result reveals that all previous analyses for this settin…
