Shape restricted statistical estimation problems have been extensively studied, with many important practical applications in signal processing, bioinformatics, and machine learning. In this paper, we propose and study a generalized nearly isotonic optimization (GNIO) model, which recovers, as special cases, many classic problems in shape constrained statistical regression, such as isotonic regression, nearly isotonic regression and unimodal regression problems. We develop an efficient and easy-to-implement dynamic programming algorithm for solving the proposed model whose recursion nature is carefully uncovered and exploited. For special $\ell_2$-GNIO problems, implementation details and the optimal ${\cal O}(n)$ running time analysis of our algorithm are discussed. Numerical experiments, including the comparisons among our approach, the powerful commercial solver Gurobi, and existing fast algorithms for solving $\ell_1$-GNIO and $\ell_2$-GNIO problems, on both simulated and real data sets, are presented to demonstrate the high efficiency and robustness of our proposed algorithm in solving large scale GNIO problems.
In this paper we study minimax and adaptation rates in general isotonic regression. For uniform deterministic and random designs in $[0,1]^d$ with $d\ge 2$ and $N(0,1)$ noise, the minimax rate for the $\ell_2$ risk is known to be bounded from below by $n^{-1/d}$ when the unknown mean function $f$ is nondecreasing and its range is bounded by a constant, while the least squares estimator (LSE) is known to nearly achieve the minimax rate up to a factor $(\log n)^γ$ where $n$ is sample size, $γ= 4$ in the lattice design and $γ= \max\{9/2, (d^2+d+1)/2 \}$ in the random design. Moreover, the LSE is …
This paper proposes a polynomial-time algorithm to construct the monotone stepwise curve that minimizes the sum of squared errors with respect to a given cloud of data points. The fitted curve is also constrained on the maximum number of steps it can be composed of and on the minimum step length. Our algorithm relies on dynamic programming and is built on the basis that said curve-fitting task can be tackled as a shortest-path type of problem. Numerical results on synthetic and realistic data sets reveal that our algorithm is able to provide the globally optimal monotone stepwise curve fit for…
Nearly optimal minimax estimator for high-dimensional sparse linear regression
我们提出了一个很好的研究统计估计问题的估计：估计的线性回归模型与软稀疏约束（$\ell_q$约束与$0<q\leq1$）在高维设置。我们首先提出了一个家庭的估计，称为投影最近邻估计，并显示，通过使用凸几何的结果，这种估计是在一个对数因子的最佳任何设计矩阵。然后利用[SIAM J. COMPUT. 36（2007）1764-1776]，我们获得了用于计算任何这样的估计任务的极大极小风险的近似算法，并且还获得了用于$\ell_1$稀疏约束的重要情况的多项式时间近似最优估计器。尽管对这个问题进行了数十年的研究，但以前只知道特殊情况下的结果。当参数半径未知时，我们也将该方法推广到自适应的情况。
This note provides upper bounds on the number of operations required to compute by value iterations a nearly optimal policy for an infinite-horizon discounted Markov decision process with a finite number of states and actions. For a given discount factor, magnitude of the reward function, and desired closeness to optimality, these upper bounds are strongly polynomial in the number of state-action pairs, and one of the provided upper bounds has the property that it is a non-decreasing function of the value of the discount factor.
We propose a novel randomized linear programming algorithm for approximating the optimal policy of the discounted Markov decision problem. By leveraging the value-policy duality and binary-tree data structures, the algorithm adaptively samples state-action-state transitions and makes exponentiated primal-dual updates. We show that it finds an $ε$-optimal policy using nearly-linear run time in the worst case. When the Markov decision process is ergodic and specified in some special data formats, the algorithm finds an $ε$-optimal policy using run time linear in the total number of state-action …
We discuss a general approach to handling "multiple hypotheses" testing in the case when a particular hypothesis states that the vector of parameters identifying the distribution of observations belongs to a convex compact set associated with the hypothesis. With our approach, this problem reduces to testing the hypotheses pairwise. Our central result is a test for a pair of hypotheses of the outlined type which, under appropriate assumptions, is provably nearly optimal. The test is yielded by a solution to a convex programming problem, so that our construction admits computationally efficient…
We give a nearly-linear time reduction that encodes any linear program as a 2-commodity flow problem with only a small blow-up in size. Under mild assumptions similar to those employed by modern fast solvers for linear programs, our reduction causes only a polylogarithmic multiplicative increase in the size of the program and runs in nearly-linear time. Our reduction applies to high-accuracy approximation algorithms and exact algorithms. Given an approximate solution to the 2-commodity flow problem, we can extract a solution to the linear program in linear time with only a polynomial factor in…
We prove that a "first-order" Sequential Quadratic Programming (SQP) algorithm for equality constrained optimization has local linear convergence with rate $(1-1/κ_R)^k$, where $κ_R$ is the condition number of the Riemannian Hessian, and global convergence with rate $k^{-1/4}$. Our analysis builds on insights from Riemannian optimization -- we show that the SQP and Riemannian gradient methods have nearly identical behavior near the constraint manifold, which could be of broader interest for understanding constrained optimization.
Mean field social optimization: feedback person-by-person optimality and the dynamic programming equation
School of Mathematics and Statistics, Carleton University（卡尔顿大学数学与统计学院）；GERAD；Department of Mathematics, National Central University（中央大学数学系）；Department of Mathematical Sciences, National Chengchi University（国立政治大学数学科学系）；Institute of Statistics, National Central University（中央大学统计研究所）
A new relaxed variant of interior point method for low-rank semidefinite programming problems is proposed in this paper. The method is a step outside of the usual interior point framework. In anticipation to converging to a low-rank primal solution, a special nearly low-rank form of all primal iterates is imposed. To accommodate such a (restrictive) structure, the first order optimality conditions have to be relaxed and are therefore approximated by solving an auxiliary least-squares problem. The relaxed interior point framework opens numerous possibilities how primal and dual approximated New…
This paper introduces a new robust interior point method analysis for semidefinite programming (SDP). This new robust analysis can be combined with either logarithmic barrier or hybrid barrier. Under this new framework, we can improve the running time of semidefinite programming (SDP) with variable size $n \times n$ and $m$ constraints up to $ε$ accuracy. We show that for the case $m = Ω(n^2)$, we can solve SDPs in $m^ω$ time. This suggests solving SDP is nearly as fast as solving the linear system with equal number of variables and constraints. This is the first result that tall dense SDP can…
In this paper we consider a class of optimization problems with a strongly convex objective function and the feasible set given by an intersection of a simple convex set with a set given by a number of linear equality and inequality constraints. A number of optimization problems in applications can be stated in this form, examples being the entropy-linear programming, the ridge regression, the elastic net, the regularized optimal transport, etc. We extend the Fast Gradient Method applied to the dual problem in order to make it primal-dual so that it allows not only to solve the dual problem, b…
We construct an algorithm, running in time $\tilde{\mathcal O}(N d + uK d)$, which is robust to outliers and heavy-tailed data and which achieves the subgaussian rate from [Lugosi, Mendelson] \begin{equation}\label{eq:intro_subgaus_rate} \sqrt{\frac{{\rm Tr}(Σ)}{N}}+\sqrt{\frac{||Σ||_{op}K}{N}} \end{equation}with probability at least $1-\exp(-c_0K)-\exp(-c_1 u)$ where $Σ$ is the covariance matrix of the informative data, $K\in\{1, \ldots, K\}$ is some parameter (number of block means) and $u>0$ is another parameter of the algorithm. This rate is achieved when $K\geq c_1 |\mathcal O|$ where $|\…
LAAS-CNRS（法国国家科学研究中心自动化、计算机工程、信号与机器人实验室）；Université de Toulouse（图卢兹大学）；IMT（图卢兹理工学院）；IRIT（图卢兹计算机研究所）
The Lipschitz constant of a network plays an important role in many applications of deep learning, such as robustness certification and Wasserstein Generative Adversarial Network. We introduce a semidefinite programming hierarchy to estimate the global and local Lipschitz constant of a multiple layer deep neural network. The novelty is to combine a polynomial lifting for ReLU functions derivatives with a weak generalization of Putinar's positivity certificate. This idea could also apply to other, nearly sparse, polynomial optimization problems in machine learning. We empirically demonstrate th…
From Sequential to Parallel: Reformulating Dynamic Programming as GPU Kernels for Large-Scale Stochastic Combinatorial Optimization
Shenzhen Research Institute of Big Data（深圳大数据研究院）；School of Data Science, The Chinese University of Hongkong（香港中文大学数据科学学院）；Shenzhen International Center for Industrial and Applied Mathematics（深圳工业与应用数学中心）；AutoKernel
A Gradient Guided Diffusion Framework for Chance Constrained Programming
School of Advanced Interdisciplinary Sciences University of Chinese Academy of Sciences（中国科学院大学先进交叉学科学院）；Department of Mathematics Sichuan University（四川大学数学学院）；Ministry of Education Key Laboratory of Mathematics and Information Networks School of Mathematical Sciences Beijing University of Posts and Telecommunications（教育部数学与信息网络重点实验室北京邮电大学数学学院）
We give the first approximation algorithm for mixed packing and covering semidefinite programs (SDPs) with polylogarithmic dependence on width. Mixed packing and covering SDPs constitute a fundamental algorithmic primitive with recent applications in combinatorial optimization, robust learning, and quantum complexity. The current approximate solvers for positive semidefinite programming can handle only pure packing instances, and technical hurdles prevent their generalization to a wider class of positive instances. For a given multiplicative accuracy of $ε$, our algorithm takes $O(\log^3(ndρ) …
Maximum Cuts and Fractional Cut Covers: A Computational Study of a Randomized Semidefinite Programming Approach
University of Waterloo（滑铁卢大学）；Institute of Mathematics, Statistics, and Computer Science, University of São Paulo（圣保罗大学数学、统计和计算机科学学院）；Center for Mathematics, Computing and Cognition, Federal University of the ABC Region（ABC地区联邦大学数学、计算与认知中心）
In this paper we show how to recover a spectral approximations to broad classes of structured matrices using only a polylogarithmic number of adaptive linear measurements to either the matrix or its inverse. Leveraging this result we obtain faster algorithms for variety of linear algebraic problems. Key results include: $\bullet$ A nearly linear time algorithm for solving the inverse of symmetric $M$-matrices, a strict superset of Laplacians and SDD matrices. $\bullet$ An $\tilde{O}(n^2)$ time algorithm for solving $n \times n$ linear systems that are constant spectral approximations of Laplac…
Following the breakthrough work of Tardos in the bit-complexity model, Vavasis and Ye gave the first exact algorithm for linear programming in the real model of computation with running time depending only on the constraint matrix. For solving a linear program (LP) $\max\, c^\top x,\: Ax = b,\: x \geq 0,\: A \in \mathbb{R}^{m \times n}$, Vavasis and Ye developed a primal-dual interior point method using a 'layered least squares' (LLS) step, and showed that $O(n^{3.5} \log (\barχ_A+n))$ iterations suffice to solve (LP) exactly, where $\barχ_A$ is a condition measure controlling the size of solu…
Sparse recovery is one of the most fundamental and well-studied inverse problems. Standard statistical formulations of the problem are provably solved by general convex programming techniques and more practical, fast (nearly-linear time) iterative methods. However, these latter "fast algorithms" have previously been observed to be brittle in various real-world settings. We investigate the brittleness of fast sparse recovery algorithms to generative model changes through the lens of studying their robustness to a "helpful" semi-random adversary, a framework which tests whether an algorithm over…
