基于交换引导的偏好学习用于从人类反馈中获得个性化强化学习
Swap-guided Preference Learning for Personalized Reinforcement Learning from Human Feedback
AI总结：
本文提出Swap-guided Preference Learning (SPL)来解决变分偏好学习中后验崩溃的问题，通过引入交换引导基础正则化、偏好逆自回归流和自适应潜在条件化来提升个性化强化学习效果。
摘要：
从人类反馈中学习强化学习（RLHF）是一种广泛用于对齐大规模AI系统与人类价值观的方法。然而，RLHF通常假设单一的通用奖励，忽略了多样化的偏好并限制了个性化。变分偏好学习（VPL）通过引入用户特定的潜在变量来解决这一问题。尽管其有潜力，我们发现VPL存在后验崩溃的问题。虽然这一现象在VAEs中很常见，但在偏好学习框架中尚未被识别。在稀疏偏好数据和过于表达的解码器下，VPL可能导致潜在变量被忽略，从而退化为单一奖励模型。为克服这一限制，我们提出了Swap-guided Preference Learning (SPL)。其核心思想是构建虚构的交换标注者，并利用其偏好的镜像性质来引导编码器。SPL引入了三个组件：（1）交换引导基础正则化，（2）偏好逆自回归流（P-IAF），（3）自适应潜在条件化。实验表明，SPL缓解了崩溃，丰富了用户特定的潜在变量，并提高了偏好预测。我们的代码和数据可在https://github.com/cobang0111/SPL获取。
英文摘要：
Reinforcement Learning from Human Feedback (RLHF) is a widely used approach to align large-scale AI systems with human values. However, RLHF typically assumes a single, universal reward, which overlooks diverse preferences and limits personalization. Variational Preference Learning (VPL) seeks to address this by introducing user-specific latent variables. Despite its promise, we found that VPL suffers from posterior collapse. While this phenomenon is well known in VAEs, it has not previously been identified in preference learning frameworks. Under sparse preference data and with overly expressive decoders, VPL may cause latent variables to be ignored, reverting to a single-reward model. To overcome this limitation, we propose Swap-guided Preference Learning (SPL). The key idea is to construct fictitious swap annotators and use the mirroring property of their preferences to guide the encoder. SPL introduces three components: (1) swap-guided base regularization, (2) Preferential Inverse Autoregressive Flow (P-IAF), and (3) adaptive latent conditioning. Experiments show that SPL mitigates collapse, enriches user-specific latents, and improves preference prediction. Our code and data are available at https://github.com/cobang0111/SPL
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
