信任、不信任或翻转：基于多专家反馈的鲁棒偏好基于强化学习
Trust, Don't Trust, or Flip: Robust Preference-Based Reinforcement Learning with Multi-Expert Feedback
AI总结：
TriTrust-PBRL通过多专家反馈联合学习奖励模型和信任参数，实现对抗性偏好下的鲁棒偏好强化学习。
摘要：
偏好基于强化学习（PBRL）通过学习成对轨迹比较来提供一种有希望的替代方案，而不是显式奖励工程。然而，现实中的偏好数据通常来自异质注释者，其可靠性各不相同；有些准确，有些嘈杂，有些系统性地对抗性。现有的PBRL方法要么将所有反馈视为同等重要，要么试图过滤掉不可靠的来源，但这两种方法在面对系统性提供错误偏好的对抗性注释者时都失败了。我们介绍了TriTrust-PBRL（TTP），一个统一的框架，从多专家偏好反馈中联合学习共享奖励模型和专家特定的信任参数。关键见解是信任参数在基于梯度的优化过程中自然演变，以正（信任）、近零（忽略）或负（翻转），使模型能够自动翻转对抗性偏好并恢复有用信号，而不是仅仅丢弃损坏的反馈。我们提供了理论分析，确立了可识别性保证和详细的梯度分析，解释了如何在训练过程中自然出现专家分离，而无需显式监督。经验上，我们在四个多样化的领域评估TTP，涵盖操纵任务（MetaWorld）和运动（DM Control）在各种损坏场景下。TTP实现了最先进的鲁棒性，在对抗性损坏下保持接近完美性能，而标准PBRL方法则彻底失败。值得注意的是，TTP在混合专家池中优于现有基线，成功学习了可靠和对抗性注释者的混合池，同时无需专家特征，仅需识别索引，并无缝集成到现有PBRL流程中。
英文摘要：
Preference-based reinforcement learning (PBRL) offers a promising alternative to explicit reward engineering by learning from pairwise trajectory comparisons. However, real-world preference data often comes from heterogeneous annotators with varying reliability; some accurate, some noisy, and some systematically adversarial. Existing PBRL methods either treat all feedback equally or attempt to filter out unreliable sources, but both approaches fail when faced with adversarial annotators who systematically provide incorrect preferences. We introduce TriTrust-PBRL (TTP), a unified framework that jointly learns a shared reward model and expert-specific trust parameters from multi-expert preference feedback. The key insight is that trust parameters naturally evolve during gradient-based optimization to be positive (trust), near zero (ignore), or negative (flip), enabling the model to automatically invert adversarial preferences and recover useful signal rather than merely discarding corrupted feedback. We provide theoretical analysis establishing identifiability guarantees and detailed gradient analysis that explains how expert separation emerges naturally during training without explicit supervision. Empirically, we evaluate TTP on four diverse domains spanning manipulation tasks (MetaWorld) and locomotion (DM Control) under various corruption scenarios. TTP achieves state-of-the-art robustness, maintaining near-oracle performance under adversarial corruption while standard PBRL methods fail catastrophically. Notably, TTP outperforms existing baselines by successfully learning from mixed expert pools containing both reliable and adversarial annotators, all while requiring no expert features beyond identification indices and integrating seamlessly with existing PBRL pipelines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
