稀有事件采样的分布匹配进化算法

中文摘要 AI 辅助

一项新颖的发现既要有用又令人惊讶：生成模型的输出若具有较低的被生成概率（即令人惊讶）且获得高奖励（即有用），则构成有用发现。全局优化可直接提高采样高奖励的概率，但通常需要更新模型权重。这种基于梯度的优化成本高昂，且无法使用功能强大的闭源模型。相反，现代搜索方法为发现而牺牲全局目标，采用具有局部奖励最大化目标的进化算法，使搜索仅聚焦于高概率样本。在本文中，我们将各种进化算法解释为近似马尔可夫链蒙特卡洛方法，这是一种无需优化即可从复杂分布中采样的方法。这一解释使得我们能够开发分布匹配进化算法（DME），这是一类无需更新权重即可从全局目标分布中采样的搜索方法。实验表明，在需要大量样本才能找到解决方案的问题上，DME 的样本效率高于现有方法。

英文摘要

A novel discovery is one which is both useful and surprising: a generative model's output is a useful discovery if it has a low probability of being generated (it's surprising) and a high reward (it's useful). Global optimization can directly increase the probability of sampling high rewards but typically requires updating model weights. Such gradient based optimization is expensive and bars using capable closed-source models. Instead, modern search methods for discovery sacrifice the global target, and use evolutionary algorithms with local reward maximizing objectives, permitting the search to focus only on high probability samples. In this paper, we interpret various evolutionary algorithms as approximate Markov Chain Monte Carlo, an optimization-free method to sample from complex distributions. This interpretation allows developing Distribution Matching Evolutionary Algorithms (DME), a class of search methods which sample from a global target distribution without updating weights. Empirically, DME has a higher sample efficiency than existing methods on problems requiring many samples to find a solution.

发表机构

• University of Oxford（牛津大学）

机构由 AI 辅助整理，请以论文原文为准。