反馈操纵正则化：实现用于模仿学习的离线智能体对齐
Feedback Manipulation Regularization: Enabling Offline Agent Alignment for Imitation Learning
AI总结：
研究聚焦强化学习智能体行为与人类价值观对齐。提出反馈操纵正则化算法，利用评估反馈校正模仿学习策略。通过改编安全体育馆环境测试，该方法能提升适应性、减少对齐错误，在有限数据下也保持稳健。
摘要：
强化学习研究日益将重点转向对齐，以确保智能体学习符合人类价值观的行为。虽然人类示范和反馈对对齐至关重要，但现有方法主要使用为语言生成的上下文博弈框架设计的多阶段管道来组合这些信号。然而，很少有工作探索这些互补输入如何能在完全顺序决策环境中作为更丰富、相互关联的信号用于单阶段离线训练。我们提出了反馈操纵正则化（FMR），这是一种与算法无关的方法，利用评估反馈作为校正信号来改善模仿学习策略的对齐。我们将安全体育馆环境改编为用于对齐评估的有原则的测试平台，证明在一系列模仿学习算法中，其适应性得到改善，对齐错误最多减少98%。即使从稀缺的对齐和无信息噪声示范中学习，FMR在有限数据情况下仍保持稳健。
英文摘要：
Reinforcement learning (RL) research has increasingly shifted focus towards alignment, ensuring agents learn behaviors adhering to human values. While human demonstrations and feedback have proven crucial for alignment, existing approaches predominantly combine these signals using multi-stage pipelines designed for the contextual bandit framing of language generation. Yet little work explores how these complementary inputs can serve as a richer, interconnected signal for single-stage offline training in fully sequential decision-making environments. We propose Feedback Manipulation Regularization (FMR), an algorithm-agnostic method that harnesses evaluative feedback as a corrective signal to improve the alignment of imitation learning policies. We adapt Safety Gymnasium environments to be a principled testbed for alignment evaluation, demonstrating improved aptitude and up to a 98\% reduction in misalignment across a range of imitation learning algorithms. FMR remains robust in limited data regimes, even when learning from scarce aligned and uninformative noisy demonstrations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
