首页/论文/2002.03755

arXiv 2002.037552020-02cs.LGmath.OCstat.ML

Compositional ADAM: An Adaptive Compositional Solver

Huawei Technologies Research and Development（华为技术研发）；University College London（伦敦大学学院）

Rasul Tutunov, Minne Li, Alexander I. Cowen-Rivers, Jun Wang, Haitham Bou-Ammar

英文摘要：

In this paper, we present C-ADAM, the first adaptive solver for compositional problems involving a non-linear functional nesting of expected values. We proof that C-ADAM converges to a stationary point in $\mathcal{O}(δ^{-2.25})$ with $δ$ being a precision parameter. Moreover, we demonstrate the importance of our results by bridging, for the first time, model-agnostic meta-learning (MAML) and compositional optimisation showing fastest known rates for deep network adaptation to-date. Finally, we validate our findings in a set of experiments from portfolio optimisation and meta-learning. Our results manifest significant sample complexity reductions compared to both standard and compositional solvers.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Solving Stochastic Compositional Optimization is Nearly as Easy as Solving Stochastic Optimization

Rensselaer Polytechnic Institute（伦斯勒理工学院）；University of California, Los Angeles（加州大学洛杉矶分校）

Stochastic compositional optimization generalizes classic (non-compositional) stochastic optimization to the minimization of compositions of functions. Each composition may introduce an additional expectation. The series of expectations may be nested. Stochastic compositional optimization is gaining popularity in applications such as reinforcement learning and meta learning. This paper presents a new Stochastically Corrected Stochastic Compositional gradient method (SCSC). SCSC runs in a single-time scale with a single loop, uses a fixed batch size, and guarantees to converge at the same rate …

2021-06-18cs.LGmath.OC

Learning compositional functions via multiplicative weight updates

Caltech（加州理工学院）；NVIDIA（英伟达）

Compositionality is a basic structural feature of both biological and artificial neural networks. Learning compositional functions via gradient descent incurs well known problems like vanishing and exploding gradients, making careful learning rate tuning essential for real-world applications. This paper proves that multiplicative weight updates satisfy a descent lemma tailored to compositional functions. Based on this lemma, we derive Madam -- a multiplicative version of the Adam optimiser -- and show that it can train state of the art neural network architectures without learning rate tuning.…

2021-01-08cs.LGcs.NAcs.NE

A Novel Convergence Analysis for Algorithms of the Adam Family

Department of Computer Science, The University of Iowa（爱荷华大学计算机科学系）；Machine Intelligence Technology, Alibaba Group（阿里巴巴集团机器智能技术）

Since its invention in 2014, the Adam optimizer has received tremendous attention. On one hand, it has been widely used in deep learning and many variants have been proposed, while on the other hand their theoretical convergence property remains to be a mystery. It is far from satisfactory in the sense that some studies require strong assumptions about the updates, which are not necessarily applicable in practice, while other studies still follow the original problematic convergence analysis of Adam, which was shown to be not sufficient to ensure convergence. Although rigorous convergence anal…

2021-12-07cs.LGmath.OC

Faster Adaptive Momentum-Based Federated Methods for Distributed Composition Optimization

Nanjing University of Aeronautics and Astronautics（南京航空航天大学）

Federated Learning is a popular distributed learning paradigm in machine learning. Meanwhile, composition optimization is an effective hierarchical learning model, which appears in many machine learning applications such as meta learning and robust learning. More recently, although a few federated composition optimization algorithms have been proposed, they still suffer from high sample and communication complexities. In the paper, thus, we propose a class of faster federated compositional optimization algorithms (i.e., MFCGD and AdaMFCGD) to solve the nonconvex distributed composition problem…

2023-03-30cs.LGmath.OC

A Swarm Variant for the Schrödinger Solver

BITS Pilani K K Birla Goa Campus（比拉理工学院果阿校区）

This paper introduces application of the Exponentially Averaged Momentum Particle Swarm Optimization (EM-PSO) as a derivative-free optimizer for Neural Networks. It adopts PSO's major advantages such as search space exploration and higher robustness to local minima compared to gradient-descent optimizers such as Adam. Neural network based solvers endowed with gradient optimization are now being used to approximate solutions to Differential Equations. Here, we demonstrate the novelty of EM-PSO in approximating gradients and leveraging the property in solving the Schrödinger equation, for the Pa…

2021-04-20cs.LGcs.NEmath.OC

A Novel Fast Exact Subproblem Solver for Stochastic Quasi-Newton Cubic Regularized Optimization

The University of Texas at Austin（德克萨斯大学奥斯汀分校）；SAS Institute（SAS研究所）

In this work we describe an Adaptive Regularization using Cubics (ARC) method for large-scale nonconvex unconstrained optimization using Limited-memory Quasi-Newton (LQN) matrices. ARC methods are a relatively new family of optimization strategies that utilize a cubic-regularization (CR) term in place of trust-regions and line-searches. LQN methods offer a large-scale alternative to using explicit second-order information by taking identical inputs to those used by popular first-order methods such as stochastic gradient descent (SGD). Solving the CR subproblem exactly requires Newton's method,…

2022-04-19cs.LGmath.OCstat.ML

更多相关论文收起更多论文

SA-Solver：用于扩散模型快速采样的随机 Adams 求解器

SA-Solver: Stochastic Adams Solver for Fast Sampling of Diffusion Models

University of Chinese Academy of Sciences（中国科学院大学）；Huawei Noah’s Ark Lab（华为诺亚实验室）；Peking University（北京大学）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）

本文提出 SA-Solver，一种改进的高效随机 Adams 方法，通过分析方差控制的扩散 SDE 和线性多步 SDE 求解器来求解扩散 SDE，在少步采样和基准数据集上实现了 SOTA 性能。

2025-06-24cs.LGstat.ML

ERA-Solver：用于扩散概率模型快速采样的误差鲁棒Adams求解器

ERA-Solver: Error-Robust Adams Solver for Fast Sampling of Diffusion Probabilistic Models

Zhejiang University（浙江大学）；Microsoft Azure Speech（微软Azure语音）；Microsoft Research Asia（微软亚洲研究院）

本文提出ERA-Solver，一种无需额外训练的误差鲁棒Adams求解器，通过Lagrange插值预测器和校正器自适应选择低误差基，在多个数据集上以仅10次网络评估实现高效高质量图像生成。

2025-11-01cs.AIcs.CVcs.LG

MAdam: 度量感知的多目标Adam

MAdam: Metric-Aware Multi-Objective Adam

Cornell Tech（康奈尔科技）；Weill Cornell Medicine（韦尔医学院）；Delft University of Technology（代尔夫特理工大学）

提出MAdam，通过偏好条件曲率预处理多目标优化中的协调方向，解决Adam与求解器之间的权重失配和几何失配问题，在多任务学习、帕累托前沿恢复等任务中一致提升性能。

2026-10-08cs.CVcs.LG

用于改进黑盒优化求解器的机器学习算法

Machine Learning Algorithms for Improving Black Box Optimization Solvers

Fakultät für Mathematik, Universität Wien（维也纳大学数学系）；Department of Computer Science, Czech Technical University（捷克技术大学计算机科学系）

本文综述 ML 与 RL 如何通过代理模型、元学习、生成模型和动态策略配置增强经典黑盒优化求解器，并梳理代表性算法与基准。

2025-09-29cs.LG

基于组合扩散的连续约束求解器

Compositional Diffusion-Based Continuous Constraint Solvers

Massachusetts Institute of Technology（麻省理工学院）；Stanford University（斯坦福大学）

本文提出Diffusion-CCSP，一种基于因子图与扩散模型能量组合的连续约束求解器，可泛化至新约束组合，并集成于任务与运动规划器以生成含离散与连续参数的长期规划。

2023-09-02cs.AIcs.LGcs.RO

Training Neural Networks using SAT solvers

Indian Institute of Technology - Kharagpur（印度理工学院卡拉格布尔分校）

We propose an algorithm to explore the global optimization method, using SAT solvers, for training a neural net. Deep Neural Networks have achieved great feats in tasks like-image recognition, speech recognition, etc. Much of their success can be attributed to the gradient-based optimisation methods, which scale well to huge datasets while still giving solutions, better than any other existing methods. However, there exist learning problems like the parity function and the Fast Fourier Transform, where a neural network using gradient-based optimisation algorithm can not capture the underlying …

2022-06-10cs.LG

Verification of Sigmoidal Artificial Neural Networks using iSAT

German Aerospace Center e.V (DLR)（德国航空航天中心（DLR））；Carl von Ossietzky University Oldenburg（奥尔登堡卡尔·冯·奥西茨基大学）

This paper presents an approach for verifying the behaviour of nonlinear Artificial Neural Networks (ANNs) found in cyber-physical safety-critical systems. We implement a dedicated interval constraint propagator for the sigmoid function into the SMT solver iSAT and compare this approach with a compositional approach encoding the sigmoid function by basic arithmetic features available in iSAT and an approximating approach. Our experimental results show that the dedicated and the compositional approach clearly outperform the approximating approach. Throughout all our benchmarks, the dedicated ap…

2022-07-14cs.AIcs.LGcs.SC

ReLoop：结构建模与行为验证用于可靠的基于LLM的优化

ReLoop: Structured Modeling and Behavioral Verification for Reliable LLM-Based Optimization

McCormick School of Engineering, Northwestern University（西北大学工程学院）；Wenzhou Buyi Pharmacy Chain Co., Ltd.（温州-buyi药链有限公司）；College of Computer Science and Artificial Intelligence, Wenzhou University（温州大学计算机科学与人工智能学院）；Department of Decision Analytics and Operations, City University of Hong Kong（香港城市大学决策分析与运营部门）；Institute of Operations Research and Analytics, National University of Singapore（新加坡国立大学运筹学与分析研究所）

ReLoop通过结构生成和行为验证机制，解决LLM生成优化代码的可行性与正确性差距问题，提升代码执行准确性和鲁棒性。

2026-10-07cs.AIcs.LGcs.SE