通过策略引导梯度搜索实现基于模型的离线优化
Offline Model-Based Optimization via Policy-Guided Gradient Search
AI总结：
本文将离线优化重构为离线强化学习问题，提出策略引导梯度搜索，使搜索策略直接从离线数据学习，并可与现有代理模型结合显著提升性能。
摘要：
离线优化是蛋白质、药物或飞机设计等许多实验工程领域中一个新兴问题，在这些领域中，通过在线实验收集评估数据成本过高或过于危险。为避免这一点，人们必须仅依据未知函数在一组固定输入上的离线评估结果来优化该函数。解决此问题的一种朴素方案是学习未知函数的代理模型，然后转而优化该代理模型。然而，这种朴素优化器在离线数据集之外的输入上，容易对代理模型产生错误的高估（这可能是由于在有偏的函数评估样本上过拟合所致）。此前应对这一挑战的方法主要集中于学习鲁棒的代理模型。然而，它们的搜索策略源自代理模型，而非实际的离线数据。为填补这一重要空白，我们通过将离线优化重构为离线强化学习问题，引入了一种新的“学习搜索”视角。我们提出的策略引导梯度搜索方法，针对由离线数据创建的给定代理模型，显式学习最佳策略。我们在多个基准上的实验结果表明，所学得的优化策略可与现有离线代理模型相结合，显著提升优化性能。
英文摘要：
Offline optimization is an emerging problem in many experimental engineering domains including protein, drug or aircraft design, where online experimentation to collect evaluation data is too expensive or dangerous. To avoid that, one has to optimize an unknown function given only its offline evaluation at a fixed set of inputs. A naive solution to this problem is to learn a surrogate model of the unknown function and optimize this surrogate instead. However, such a naive optimizer is prone to erroneous overestimation of the surrogate (possibly due to over-fitting on a biased sample of function evaluation) on inputs outside the offline dataset. Prior approaches addressing this challenge have primarily focused on learning robust surrogate models. However, their search strategies are derived from the surrogate model rather than the actual offline data. To fill this important gap, we introduce a new learning-to-search perspective for offline optimization by reformulating it as an offline reinforcement learning problem. Our proposed policy-guided gradient search approach explicitly learns the best policy for a given surrogate model created from the offline data. Our empirical results on multiple benchmarks demonstrate that the learned optimization policy can be combined with existing offline surrogates to significantly improve the optimization performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
