We consider variants of the classical Frank-Wolfe algorithm for constrained smooth convex minimization, that instead of access to the standard oracle for minimizing a linear function over the feasible set, have access to an oracle that can find an extreme point of the feasible set that is closest in Euclidean distance to a given vector. We first show that for many feasible sets of interest, such an oracle can be implemented with the same complexity as the standard linear optimization oracle. We then show that with such an oracle we can design new Frank-Wolfe variants which enjoy significantly improved complexity bounds in case the set of optimal solutions lies in the convex hull of a subset of extreme points with small diameter (e.g., a low-dimensional face of a polytope). In particular, for many $0\text{--}1$ polytopes, under quadratic growth and strict complementarity conditions, we obtain the first linearly convergent variant with rate that depends only on the dimension of the optimal face and not on the ambient dimension.
We study Frank-Wolfe algorithms - standard, pairwise, and away-steps - for efficient optimization of Dominant Set Clustering. We present a unified and computationally efficient framework to employ the different variants of Frank-Wolfe methods, and we investigate its effectiveness via several experimental studies. In addition, we provide explicit convergence rates for the algorithms in terms of the so-called Frank-Wolfe gap. The theoretical analysis has been specialized to Dominant Set Clustering and covers consistently the different variants.
The Frank-Wolfe algorithm has regained much interest in its use in structurally constrained machine learning applications. However, one major limitation of the Frank-Wolfe algorithm is the slow local convergence property due to the zig-zagging behavior. We observe the zig-zagging phenomenon in the Frank-Wolfe method as an artifact of discretization, and propose multistep Frank-Wolfe variants where the truncation errors decay as $O(Δ^p)$, where $p$ is the method's order. This strategy "stabilizes" the method, and allows tools like line search and momentum to have more benefits. However, our res…
Scalable Frank-Wolfe on Generalized Self-concordant Functions via Simple Steps
Georgia Institute of Technology（佐治亚理工学院）；Zuse Institute Berlin（柏林祖斯研究所）；Univ. Grenoble Alpes（格勒诺布尔大学）；Inria（法国国家信息与自动化研究所）；Technische Universität Berlin（柏林工业大学）
Boosted Stochastic Frank-Wolfe for Constrained Nonconvex Optimization
National University of Singapore（新加坡国立大学）；School of Mathematics and Computer Science, Iran University of Science and Technology（伊朗科技大学数学与计算机科学学院）；CentraleSupélec, Université Paris-Saclay, Inria（中央理工-高等电力学院，巴黎萨克雷大学，法国国家信息与自动化研究所）
Interpretable Neural Networks with Frank-Wolfe: Sparse Relevance Maps and Relevance Orderings
Institut für Mathematik, Technische Universität Berlin（柏林工业大学数学研究所）；Department for AI in Society, Science, and Technology, Zuse Institute Berlin（柏林楚泽研究所社会、科学与技术人工智能系）
We introduce regularized Frank-Wolfe, a general and effective algorithm for inference and learning of dense conditional random fields (CRFs). The algorithm optimizes a nonconvex continuous relaxation of the CRF inference problem using vanilla Frank-Wolfe with approximate updates, which are equivalent to minimizing a regularized energy function. Our proposed method is a generalization of existing algorithms such as mean field or concave-convex procedure. This perspective not only offers a unified analysis of these algorithms, but also allows an easy way of exploring different variants that pote…
We propose several variants of the Frank-Wolfe algorithm to minimize a sum of functions. The main proposed algorithm is inspired from the dual averaging scheme of Nesterov adapted for Frank Wolfe in a stochastic setting. A distributed version of this scheme is also suggested. Additionally, we propose a Frank-Wolfe variant based on incremental gradient techniques. The convergence rates for all the proposed algorithms are established. The performance is studied on least squares regression and multinomial classification.
Technion - Israel Institute of Technology（以色列理工学院）
Tyler's M-estimator is a well known procedure for robust and heavy-tailed covariance estimation. Tyler himself suggested an iterative fixed-point algorithm for computing his estimator however, it requires super-linear (in the size of the data) runtime per iteration, which maybe prohibitive in large scale. In this work we propose, to the best of our knowledge, the first Frank-Wolfe-based algorithms for computing Tyler's estimator. One variant uses standard Frank-Wolfe steps, the second also considers \textit{away-steps} (AFW), and the third is a \textit{geodesic} version of AFW (GAFW). AFW prov…
University of California, Berkeley（加州大学伯克利分校）；ETH Zurich（苏黎世联邦理工学院）；SumUp Analytics（SumUp分析公司）；Max-Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；MIT Sloan School of Management（麻省理工学院斯隆管理学院）；Google Research（谷歌研究院）
We propose a novel Stochastic Frank-Wolfe (a.k.a. conditional gradient) algorithm for constrained smooth finite-sum minimization with a generalized linear prediction/structure. This class of problems includes empirical risk minimization with sparse, low-rank, or other structured constraints. The proposed method is simple to implement, does not require step-size tuning, and has a constant per-iteration cost that is independent of the dataset size. Furthermore, as a byproduct of the method we obtain a stochastic estimator of the Frank-Wolfe gap that can be used as a stopping criterion. Depending…
Accelerating Frank-Wolfe via Averaging Step Directions
Frank-Wolfe方法因其快速的每次迭代复杂度而成为稀疏约束优化中的一种流行方法。然而，折衷是其最坏情况下整体收敛相对较慢，并且重要的是，从根本上慢于其流率--也就是说，收敛速度被离散化误差抑制。在这项工作中，我们考虑修改后的弗兰克-沃尔夫的步骤方向是一个简单的加权平均过去的甲骨文调用。这种方法需要非常少的内存和计算开销，并证明衰减这个离散化误差项。数值上，我们表明，这种方法提高了收敛速度在几个问题，特别是在稀疏流形已被检测。理论上，我们证明了该方法具有$O(1/k^p)$的整体收敛速度，其中$0< p < 1$ ;流形识别后，该速度加快到$O(1/k^{3p/2})$。我们还观察到，该方法从很早的阶段就实现了这种加速率，这表明这一系列方法的加速模式很有前途。
This paper uncovers a simple but rather surprising connection: it shows that the well-known convex-concave procedure (CCCP) and its generalization to constrained problems are both special cases of the Frank-Wolfe (FW) method. This connection not only provides insight of deep (in our opinion) pedagogical value, but also transfers the recently discovered convergence theory of nonconvex Frank-Wolfe methods immediately to CCCP, closing a long-standing gap in its non-asymptotic convergence theory. We hope the viewpoint uncovered by this paper spurs the transfer of other advances made for FW to both…
Massachusetts Institute of Technology（麻省理工学院）；JP Morgan AI Research（摩根大通人工智能研究院）
Hawkes processes have recently risen to the forefront of tools when it comes to modeling and generating sequential events data. Multidimensional Hawkes processes model both the self and cross-excitation between different types of events and have been applied successfully in various domain such as finance, epidemiology and personalized recommendations, among others. In this work we present an adaptation of the Frank-Wolfe algorithm for learning multidimensional Hawkes processes. Experimental results show that our approach has better or on par accuracy in terms of parameter estimation than other…
Frank-Wolfe Algorithms for (L0, L1)-smooth functions
Moscow Institute of Physics and Technology（莫斯科物理技术学院）；Innopolis University（伊诺波利斯大学）；Adyghe State University（阿迪亚耶州立大学）；V. I. Vernadsky Crimean Federal University（弗拉基米尔·伊万诺维奇·维尔纳茨基克里米亚联邦大学）
