ProteinOPD：迈向蛋白质设计的有效和高效偏好对齐
ProteinOPD: Towards Effective and Efficient Preference Alignment for Protein Design
AI总结：
本文提出ProteinOPD框架，通过结合On-Policy Distillation方法，有效平衡多目标偏好，提升蛋白质设计效率与稳定性，实验显示在目标偏好上表现优异且训练速度更快。
摘要：
设计具有预期功能或性质的蛋白质代表了合成生物学和药物发现中的核心目标。最近的蛋白质语言模型（PLMs）进展使得生成高度可设计的蛋白质序列成为可能，而偏好对齐提供了一种引导设计向期望功能和性质发展的有希望的方法。然而，它们常常触发预训练知识的灾难性遗忘，降低基本设计性并未能平衡多个竞争性目标。为了解决这些问题，我们受到On-Policy Distillation（OPD）的启发，这是一种先进的后训练方法，以其模式寻求性质而闻名，能够通过缓解灾难性遗忘。在本工作中，我们提出了ProteinOPD，一个多目标偏好对齐框架，能够在保持PLMs固有设计性的同时有效平衡多个偏好目标。ProteinOPD将预训练的PLM适配为偏好特定的教师，并通过学生自身轨迹上的令牌级OPD将他们的知识蒸馏到共享的学生中。在此过程中，学生被对齐到加权教师的唯一标准化几何共识，同时在冲突下确保有界优化。这填补了OPD在多目标/教师对齐中的空白。广泛的实验表明，ProteinOPD在目标偏好目标上实现了显著的收益，而不会牺牲设计性，与基于强化学习的对齐竞争对手相比，训练速度提升了8倍。
英文摘要：
Designing proteins with desired functions or properties represents a core goal in synthetic biology and drug discovery. Recent advances in protein language models (PLMs) have enabled the generation of highly designable protein sequences, while preference alignment provides a promising way to steer designs toward desired functions and properties. Nevertheless, they often trigger catastrophic forgetting of pretrained knowledge, degrading basic designability and failing to balance multiple competing objectives. To address these issues, we draw inspiration from On-Policy Distillation (OPD), an advanced post-training method renowned for mitigating catastrophic forgetting through its mode-seeking nature. In this work, we propose ProteinOPD, a multi-objective preference alignment framework that can effectively balance multiple preference objectives while maintaining the inherent designability of PLMs. ProteinOPD adapts a pretrained PLM into preference-specific teachers and distills their knowledge into a shared student via token-level OPD on the student's own trajectories. During this process, the student is aligned to a unique normalized geometric consensus of weighted teachers while ensuring bounded optimization under conflicts. This bridges the gap for OPD in multi-objective/teacher alignment. Extensive experiments show that ProteinOPD achieves substantial gains on target preference objectives without compromising the designability, with an 8x training speedup over RL-based alignment competitors.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
