FLoRA: 基于低秩奖励函数风格适配的样本高效偏好强化学习
FLoRA: Sample-Efficient Preference-based RL via Low-Rank Style Adaptation of Reward Functions
AI总结：
针对低偏好数据场景下预训练机器人适配的灾难性奖励遗忘问题，提出通过低秩矩阵增强原始奖励模型的FLoRA方法，实现了对人类偏好的高效适配，在模拟和真实任务中均有效。
摘要：
基于偏好的强化学习（PbRL）是预训练机器人行为风格适配的合适方法：在让机器人策略遵循人类用户偏好的同时，仍能执行原始任务。然而，在机器人学中为适配过程收集偏好往往具有挑战性且耗时。本研究探索了低偏好数据场景下预训练机器人的适配问题。我们发现，在此场景下，近期的适配方法存在灾难性奖励遗忘（CRF）问题：更新后的奖励模型过拟合新偏好，导致智能体无法执行原始任务。为缓解CRF，我们提出通过少量负责建模偏好适配的参数（低秩矩阵）增强原始奖励模型。评估表明，我们的方法能在模拟基准任务和多个真实世界机器人任务中高效且有效地调整机器人行为以匹配人类偏好。
英文摘要：
Preference-based reinforcement learning (PbRL) is a suitable approach for style adaptation of pre-trained robotic behavior: adapting the robot's policy to follow human user preferences while still being able to perform the original task. However, collecting preferences for the adaptation process in robotics is often challenging and time-consuming. In this work we explore the adaptation of pre-trained robots in the low-preference-data regime. We show that, in this regime, recent adaptation approaches suffer from catastrophic reward forgetting (CRF), where the updated reward model overfits to the new preferences, leading the agent to become unable to perform the original task. To mitigate CRF, we propose to enhance the original reward model with a small number of parameters (low-rank matrices) responsible for modeling the preference adaptation. Our evaluation shows that our method can efficiently and effectively adjust robotic behavior to human preferences across simulation benchmark tasks and multiple real-world robotic tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
