通过能量排名对齐实现连续反馈下的Transformer对齐
Aligning Transformers with Continuous Feedback via Energy Rank Alignment
AI总结：
针对化学空间搜索中的分子生成问题，提出能量排名对齐（ERA）算法，利用显式奖励函数优化自回归策略，无需强化学习，可稳健生成具有指定性质的分子和蛋白质序列。
摘要：
在化学空间中搜索是一个极其具有挑战性的问题，因为可能的分子的数量随原子数量呈组合式增长。基于化合物数据库训练的大型自回归模型已经产生了强大的生成器，但我们仍然缺乏生成具有所需性质的分子的稳健策略。这个分子搜索问题与大型语言模型的“对齐”问题非常相似，尽管对于许多化学任务，我们有一个具体且易于评估的奖励函数。在这里，我们介绍了一种称为能量排名对齐（ERA）的算法，它利用显式的奖励函数来产生一个基于梯度的目标，我们用它来优化自回归策略。我们在理论上表明，该算法与近端策略优化（PPO）和直接偏好优化（DPO）密切相关，但其最小化器收敛到一个理想的吉布斯-玻尔兹曼分布，其中奖励扮演能量函数的角色。此外，该算法高度可扩展，不需要强化学习，并且在每次配对的偏好观察数量较少时，相对于DPO表现良好。我们部署这种方法来对齐分子Transformer和蛋白质语言模型，分别生成具有外部指定性质的分子和蛋白质序列，并发现它能稳健地做到这一点，在化学空间的不同部分进行搜索。
英文摘要：
Searching through chemical space is an exceptionally challenging problem because the number of possible molecules grows combinatorially with the number of atoms. Large, autoregressive models trained on databases of chemical compounds have yielded powerful generators, but we still lack robust strategies for generating molecules with desired properties. This molecular search problem closely resembles the "alignment" problem for large language models, though for many chemical tasks we have a specific and easily evaluable reward function. Here, we introduce an algorithm called energy rank alignment (ERA) that leverages an explicit reward function to produce a gradient-based objective that we use to optimize autoregressive policies. We show theoretically that this algorithm is closely related to proximal policy optimization (PPO) and direct preference optimization (DPO), but has a minimizer that converges to an ideal Gibbs-Boltzmann distribution with the reward playing the role of an energy function. Furthermore, this algorithm is highly scalable, does not require reinforcement learning, and performs well relative to DPO when the number of preference observations per pairing is small. We deploy this approach to align molecular transformers and protein language models to generate molecules and protein sequences, respectively, with externally specified properties and find that it does so robustly, searching through diverse parts of chemical space.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
