PAPA：在线个性化主动偏好对齐
PAPA: Online Personalized Active Preference Alignment
AI总结：
提出PAPA方法，无需参数化奖励模型，通过实时用户反馈直接优化扩散模型，实现高效的偏好对齐，并基于理论提出加速微调的EPAPA策略。
摘要：
扩散模型在建模复杂数据分布（包括图像和文本）方面非常有效。然而，在个性化推荐系统等应用中，目标通常转向建模分布中最大化用户偏好的特定区域——这些偏好最初未知，但通过交互式反馈逐渐揭示。这自然可以表述为一个强化学习问题，其目标是微调扩散模型以最大化基于偏好的奖励函数。然而，主要挑战在于学习参数化的奖励模型，这通常需要大规模的偏好数据——在实践中往往不可行。在这项工作中，我们引入了个性化主动偏好对齐（PAPA），这是一种新颖的方法，通过直接使用实时用户反馈优化扩散模型，绕过了对参数化奖励模型的需求。PAPA 实现了反馈高效的偏好对齐，其灵感来自变分推理框架。我们通过跨不同类别条件和细粒度对齐任务的广泛实验和消融研究证明了 PAPA 的有效性。此外，基于理论见解，我们提出了一种增强的微调策略，称为 EPAPA，它需要更少的计算预算并加速微调过程，进一步提升了 PAPA 在实际部署中的适用性。我们的代码在此 https URL 公开提供。
英文摘要：
Diffusion models are highly effective at modeling complex data distributions, including images and text. However, in applications like personalized recommender systems, the objective often shifts to modeling specific regions of the distribution that maximize user preferences-initially unknown but gradually uncovered through interactive feedback. This can naturally be framed as a reinforcement learning problem, where the goal is to fine-tune a diffusion model to maximize a reward function based on preferences. However, the main challenge lies in learning a parameterized reward model, which typically requires large-scale preference data-something that is often not feasible in practice. In this work, we introduce Personalized Active Preference Alignment PAPA, a novel method that bypasses the requirement for a parametrized reward model by directly optimizing the diffusion model using real-time user feedback. PAPA enables feedback-efficient preference alignment, drawing inspiration from the variational inference framework. We demonstrate PAPA's effectiveness through extensive experiments and ablation studies across diverse class-conditioned and fine-grained alignment tasks. Additionally, based on theoretical insights, we propose an enhanced fine-tuning strategy, referred to as EPAPA, that requires less computational budget and accelerates the fine-tuning process, further boosting PAPA's suitability for real-world deployment. Our code is made publicly available at https://github.com/NasikNafi/papa.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
