arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.19733cs.LGmath.OC
收藏

理解展开的诅咒

Understanding the Curse of Unrolling

Saarland University(萨尔兰大学);Friedrich-Alexander University(弗赖堡-艾森纳赫大学)

Sheheryar Mehmood, Florian Knoll, Peter Ochs

AI总结:

本研究分析了算法展开中出现的'展开诅咒'现象,揭示了其成因,并提出通过提前截断和预热启动来缓解该问题的方法。

摘要:

算法展开在机器学习中无处不在,特别是在超参数优化和元学习中,通过迭代算法对解映射的雅可比矩阵进行求导。尽管在合适条件下展开能产生渐近正确的雅可比矩阵,但最近的研究表明,导数迭代可能最初会偏离真实的雅可比矩阵,这种现象被称为展开的诅咒。在本工作中,我们提供了一个非渐近分析,解释了这种行为的起源,并识别了支配它的算法因素。我们证明,提前截断导数计算的早期迭代可以缓解这种诅咒,同时减少内存需求。最后,我们展示在双水平优化中使用预热启动自然地诱导了一种隐含的截断形式,提供了一种实用的解决方案。我们的理论发现通过在代表性例子上的数值实验得到了支持。

英文摘要:

Algorithm unrolling is ubiquitous in machine learning, particularly in hyperparameter optimization and meta-learning, where Jacobians of solution mappings are computed by differentiating through iterative algorithms. Although unrolling is known to yield asymptotically correct Jacobians under suitable conditions, recent work has shown that the derivative iterates may initially diverge from the true Jacobian, a phenomenon known as the curse of unrolling. In this work, we provide a non-asymptotic analysis that explains the origin of this behavior and identifies the algorithmic factors that govern it. We show that truncating early iterations of the derivative computation mitigates the curse while simultaneously reducing memory requirements. Finally, we demonstrate that warm-starting in bilevel optimization naturally induces an implicit form of truncation, providing a practical remedy. Our theoretical findings are supported by numerical experiments on representative examples.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑