Leveraging machine learning to facilitate the optimization process is an emerging field that holds the promise to bypass the fundamental computational bottleneck caused by classic iterative solvers in critical applications requiring near-real-time optimization. The majority of existing approaches focus on learning data-driven optimizers that lead to fewer iterations in solving an optimization. In this paper, we take a different approach and propose to replace the iterative solvers altogether with a trainable parametric set function, that outputs the optimal arguments/parameters of an optimization problem in a single feed forward. We denote our method as Learning to Optimize the Optimization Process (LOOP). We show the feasibility of learning such parametric (set) functions to solve various classic optimization problems including linear/nonlinear regression, principal component analysis, transport-based coreset, and quadratic programming in supply management applications. In addition, we propose two alternative approaches for learning such parametric functions, with and without a solver in the LOOP. Finally, through various numerical experiments, we show that the trained solvers could be orders of magnitude faster than the classic iterative solvers while providing near optimal solutions.
Teaching Transformers to Solve Combinatorial Problems through Efficient Trial & Error
National Technical University of Athens(希腊雅典国家技术大学);National and Kapodistrian University of Athens(希腊雅典国家与卡波迪斯特里亚大学);Archimedes, Athena Research Center(阿提卡研究中心阿基米德)
Teaching Algorithmic Reasoning via In-context Learning
Mila(米拉研究所);Université de Montreal(蒙特利尔大学);Google Research(谷歌研究院)
大语言模型(LLM)通过扩展模型和数据大小,显示出越来越多的上下文学习能力。尽管取得了这些进展,LLM仍然无法解决算法推理问题。虽然为最终答案提供了一个基本原理,导致了多步推理问题的进一步改进,但Anil et al. 2022表明,即使是简单的算法推理任务,如奇偶校验,也远未解决。在这项工作中,我们确定并研究了成功教授算法推理的四个关键阶段:(1)将算法制定为技能,(2)同时教授多种技能(技能积累),(3)教授如何组合技能(技能组合)和(4)教授如何使用技能作为工具。我们表明,它是可以教算法推理LLMs通过上下文学习,我们称之为算法提示。我们评估我们的方法在各种算术和定量推理任务,并表现出显着的提升性能比现有的提示技术。特别是,对于长奇偶校验、加法、乘法和减法,与最佳可用基线相比,我们分别实现了约10倍、9倍、5倍和2倍的误差减少。
Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);University of Cambridge(剑桥大学);Technical University of Munich(慕尼黑工业大学);University of Tübingen(蒂宾根大学);Mila, Université de Montréal(蒙特利尔大学米拉人工智能研究所);The Alan Turing Institute(艾伦·图灵研究所)
Jilin University(吉林大学);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);University of Cambridge(剑桥大学);A*STAR(新加坡科技研究局);Hong Kong University of Science and Technology(香港科技大学)