arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2211.08478eess.SYcs.SY
收藏

ECCO:等效电路控制优化

ECCO: Equivalent Circuit Controlled Optimization

Aayushya Agarwal, Carmel Fiscko, Soummya Kar, Larry Pileggi, Bruno Sinopoli

摘要:

我们提出了一种求解无约束缩放梯度流问题的自适应优化算法,该算法通过控制优化轨迹形状和离散化步长来实现快速收敛。在广泛的一类尺度函数,我们建立收敛的平滑目标函数的临界点,同时证明其灵活性和鲁棒性方面的超参数调整所提出的方法。首先,我们证明了在正则性条件下分量尺度梯度流收敛到临界点。我们表明,这种控制梯度流动力学是等效的电路的瞬态响应,允许电路理论的概念来解决这个问题。基于这种等价性,我们开发了两种基于最小化电路中存储的电荷的优化轨迹控制方案:使用真正的Hessian的二阶方法和仅用梯度信息近似优化轨迹的替代一阶方法。虽然控制方案是从电路的概念,不需要电路知识来实现的算法。为了找到临界点的值,我们提出了一个时间步长搜索例程的前向欧拉离散控制的局部截断误差,一种方法,从电路模拟的想法。在仿真中,我们发现,轨迹控制优于不受控制的梯度流,和错误意识的离散化优于线搜索与Armijo条件。我们的算法进行评估凸和非凸测试功能,包括神经网络,收敛速度相当于或超过亚当。

英文摘要:

We propose an adaptive optimization algorithm for solving unconstrained scaled gradient flow problems that achieves fast convergence by controlling the optimization trajectory shape and the discretization step sizes. Under a broad class of scaling functions, we establish convergence of the proposed approach to critical points of smooth objective functions, while demonstrating its flexibility and robustness with respect to hyperparameter tuning. First, we prove convergence of component-wise scaled gradient flow to a critical point under regularity conditions. We show that this controlled gradient flow dynamics is equivalent to the transient response of an electrical circuit, allowing for circuit theory concepts to solve the problem. Based on this equivalence, we develop two optimization trajectory control schemes based on minimizing the charge stored in the circuit: a second order method that uses the true Hessian and an alternate first order method that approximates the optimization trajectory with only gradient information. While the control schemes are derived from circuit concepts, no circuit knowledge is needed to implement the algorithms. To find the value of the critical point, we propose a time step search routine for Forward Euler discretization that controls the local truncation error, a method adapted from circuit simulation ideas. In simulation we find that the trajectory control outperforms uncontrolled gradient flow, and the error-aware discretization out-performs line search with the Armijo condition. Our algorithms are evaluated on convex and non-convex test functions, including neural networks, with convergence speeds comparable to or exceeding Adam.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑