Classifiers that are linear in their parameters, and trained by optimizing a convex loss function, have predictable behavior with respect to changes in the training data, initial conditions, and optimization. Such desirable properties are absent in deep neural networks (DNNs), typically trained by non-linear fine-tuning of a pre-trained model. Previous attempts to linearize DNNs have led to interesting theoretical insights, but have not impacted the practice due to the substantial performance gap compared to standard non-linear optimization. We present the first method for linearizing a pre-trained model that achieves comparable performance to non-linear fine-tuning on most of real-world image classification tasks tested, thus enjoying the interpretability of linear models without incurring punishing losses in performance. LQF consists of simple modifications to the architecture, loss function and optimization typically used for classification: Leaky-ReLU instead of ReLU, mean squared loss instead of cross-entropy, and pre-conditioning using Kronecker factorization. None of these changes in isolation is sufficient to approach the performance of non-linear fine-tuning. When used in combination, they allow us to reach comparable performance, and even superior in the low-data regime, while enjoying the simplicity, robustness and interpretability of linear-quadratic optimization.
An exponential mechanism based on quadratic approximations for fine-tuning machine learning models with privacy guarantees
Computer Science and Mathematics Division, Oak Ridge National Laboratory（橡树岭国家实验室计算机科学与数学 division）；Computational Science and Engineering Division, Oak Ridge National Laboratory（橡树岭国家实验室计算科学与工程 division）；HPC Department, Cineca（Cineca 高性能计算部）
National University of Defense Technology（国防科技大学）；Tsinghua University（清华大学）；University of Utah（犹他大学）
Heavy ball momentum is crucial in accelerating (stochastic) gradient-based optimization algorithms for machine learning. Existing heavy ball momentum is usually weighted by a uniform hyperparameter, which relies on excessive tuning. Moreover, the calibrated fixed hyperparameter may not lead to optimal performance. In this paper, to eliminate the effort for tuning the momentum-related hyperparameter, we propose a new adaptive momentum inspired by the optimal choice of the heavy ball momentum for quadratic optimization. Our proposed adaptive heavy ball momentum can improve stochastic gradient de…
Reinforcement learning for adaptive interior point methods in convex quadratic programming
Department of Aerospace Engineering, Institute of Applied Mathematics and Scientific Computing, University of the Bundeswehr Munich（航空航天工程系、应用数学与科学计算研究所、联邦国防军大学穆尔尼奇分校）
Classification as a supervised learning concept is an important content in machine learning. It aims at categorizing a set of data into classes. There are several commonly-used classification methods nowadays such as k-nearest neighbors, random forest, and support vector machine. Each of them has its own pros and cons, and none of them is invincible for all kinds of problems. In this thesis, we focus on Quadratic Multiform Separation (QMS), a classification method recently proposed by Michael Fan et al. (2019). Its fresh concept, rich mathematical structure, and innovative definition of loss f…
Data-Driven Exploration for a Class of Continuous-Time Indefinite Linear--Quadratic Reinforcement Learning Problems
Department of Industrial Engineering and Operations Research, Columbia University（工业工程与运营管理系，哥伦比亚大学）；Data Science Institute, Columbia University（数据科学研究所，哥伦比亚大学）
针对连续时间随机 LQ 控制的无模型 RL，本文提出由 critic 与 actor 自适应调节熵正则化和策略方差的数据驱动探索机制，以少量调参实现匹配已知最优固定调度结果的次线性遗憾，并在实验中加速收敛、降低遗憾。
Princeton University（普林斯顿大学）；ETH Zurich（苏黎世联邦理工学院）；University of California, Berkeley（加州大学伯克利分校）
First-order methods for quadratic optimization such as OSQP are widely used for large-scale machine learning and embedded optimal control, where many related problems must be rapidly solved. These methods face two persistent challenges: manual hyperparameter tuning and convergence time to high-accuracy solutions. To address these, we explore how Reinforcement Learning (RL) can learn a policy to tune parameters to accelerate convergence. In experiments with well-known QP benchmarks we find that our RL policy, RLQP, significantly outperforms state-of-the-art QP solvers by up to 3x. RLQP generali…
University of Toronto（多伦多大学）；Vector Institute（向量学院）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Standard first-order stochastic optimization algorithms base their updates solely on the average mini-batch gradient, and it has been shown that tracking additional quantities such as the curvature can help de-sensitize common hyperparameters. Based on this intuition, we explore the use of exact per-sample Hessian-vector products and gradients to construct optimizers that are self-tuning and hyperparameter-free. Based on a dynamics model of the gradient, we derive a process which leads to a curvature-corrected, noise-adaptive online gradient estimate. The smoothness of our updates makes it mor…
Choosing the right parameters for optimization algorithms is often the key to their success in practice. Solving this problem using a learning-to-learn approach -- using meta-gradient descent on a meta-objective based on the trajectory that the optimizer generates -- was recently shown to be effective. However, the meta-optimization problem is difficult. In particular, the meta-gradient can often explode/vanish, and the learned optimizer may not have good generalization performance if the meta-objective is not chosen carefully. In this paper we give meta-optimization guarantees for the learnin…
University of Bojnord（博季努尔德大学）；Charles University（查理大学）；University of Minnesota（明尼苏达大学）；Northeastern University（东北大学）
In binary classification, kernel-free linear or quadratic support vector machines are proposed to avoid dealing with difficulties such as finding appropriate kernel functions or tuning their hyper-parameters. Furthermore, Universum data points, which do not belong to any class, can be exploited to embed prior knowledge into the corresponding models so that the generalization performance is improved. In this paper, we design novel kernel-free Universum quadratic surface support vector machine models. Further, we propose the L1 norm regularized version that is beneficial for detecting potential …
QuadEnhancer: Leveraging Quadratic Transformations to Enhance Deep Neural Networks
School of Science and Engineering（科学与工程学院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen International Center for Industrial and Applied Mathematics（深圳国际工业与应用数学中心）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；School of Data Science（数据科学学院）
CNRS - IRIT（法国国家科学研究中心 - 图卢兹计算机研究所）；Université de Toulouse（图卢兹大学）；Toulouse School of Economics（图卢兹经济学院）；DEEL, IRT Saint Exupery（DEEL，圣埃克苏佩里研究所）
In view of a direct and simple improvement of vanilla SGD, this paper presents a fine-tuning of its step-sizes in the mini-batch case. For doing so, one estimates curvature, based on a local quadratic model and using only noisy gradient approximations. One obtains a new stochastic first-order method (Step-Tuned SGD), enhanced by second-order information, which can be seen as a stochastic version of the classical Barzilai-Borwein method. Our theoretical results ensure almost sure convergence to the critical set and we provide convergence rates. Experiments on deep residual network training illu…
Korea Advanced Institute of Science and Technology（韩国科学技术院）；LG AI Research（LG人工智能研究院）
Pre-trained representation is one of the key elements in the success of modern deep learning. However, existing works on continual learning methods have mostly focused on learning models incrementally from scratch. In this paper, we explore an alternative framework to incremental learning where we continually fine-tune the model from a pre-trained representation. Our method takes advantage of linearization technique of a pre-trained neural network for simple and effective continual learning. We show that this allows us to design a linear model where quadratic parameter regularization method is…
Sequential QCQP for Bilevel Optimization with Line Search
Department of Electrical and Computer Engineering at Johns Hopkins University（约翰霍普金斯大学电气与计算机工程系）；Department of Systems and Industrial Engineering, The University of Arizona（亚利桑那大学系统与工业工程系）
Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuning
Center for Data Science, Peking University（北京大学数据科学中心）；Yau Mathematical Sciences Center, Tsinghua University（清华大学丘成桐数学科学中心）；Beijing International Center for Mathematical Research, Peking University（北京大学北京国际数学研究中心）；Department of Mathematics, University of California, Berkeley（美国加州大学伯克利分校数学系）；Center for Machine Learning Research and Changsha Institute for Computing and Digital Economy, Peking University（北京大学机器学习研究中心和长沙计算与数字经济研究院）
Aristotle University of Thessaloniki（塞萨洛尼基亚里士多德大学）
Language Models are the underpin of all modern Natural Language Processing (NLP) tasks. The introduction of the Transformers architecture has contributed significantly into making Language Modeling very effective across many NLP task, leading to significant advancements in the field. However, Transformers come with a big computational cost, which grows quadratically with respect to the input length. This presents a challenge as to understand long texts requires a lot of context. In this paper, we propose a Fine-Tuning framework, named CoreLM, that extends the architecture of current Pretrained…
TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordination
Department of Electrical \& Computer Engineering, University of Arizona；Engineering College, Soochow University；INSAIT, Sofia University "St. Kliment Ohridski"；Department of Electrical；Computer Engineering, Stony Brook University
International Laboratory on Learning Systems（国际学习系统实验室）；Université Paris-Saclay（巴黎-萨克雷大学）；Ecole de technologie superieur（高等技术学院）；MILA（MILA（蒙特利尔学习算法研究所））
LoKO: Low-Rank Kalman Optimizer for Online Fine-Tuning of Large Models
Department of Computer Science（计算机科学系）；The University of Manchester（曼彻斯特大学）；Department of Computer Science and Technologies（计算机科学与技术系）；University of Cambridge（剑桥大学）
City University of Hong Kong（香港城市大学）；Harbin Institute of Technology（哈尔滨工业大学）
Collaborative filtering (CF) is widely searched in recommendation with various types of solutions. Recent success of Graph Convolution Networks (GCN) in CF demonstrates the effectiveness of modeling high-order relationships through graphs, while repetitive graph convolution and iterative batch optimization limit their efficiency. Instead, item similarity models attempt to construct direct relationships through efficient interaction encoding. Despite their great performance, the growing item numbers result in quadratic growth in similarity modeling process, posing critical scalability problems.…
Control Theoretic Approach to Fine-Tuning and Transfer Learning
Coordinated Science Laboratory, University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校协调科学实验室）；The School of Automation, Beijing Institute of Technology（北京理工大学自动化学院）
Differentiable Quadratic Optimization For The Maximum Independent Set Problem
University of Michigan, Ann Arbor（密歇根大学安娜堡分校）；Michigan State University（密歇根州立大学）；Ohio State University（俄亥俄州立大学）；University of Maryland, College Park（马里兰大学学院市分校）；University of Colorado, Boulder（科罗拉多大学博尔德分校）
本文提出 pCQO-MIS，通过融合最大团项的可微二次优化、并行动量梯度下降及理论驱动的 MIS 检查来求解最大独立集，在规模、运行时间和数据依赖方面优于多类方法。
