arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.13683cs.RO
收藏

PrefMMT:利用多模态Transformer在基于偏好的强化学习中建模人类偏好

PrefMMT: Modeling Human Preferences in Preference-based Reinforcement Learning with Multimodal Transformers

College of Mechanical and Electrical Engineering, Beijing University of Chemical Technology(北京化工大学机电工程学院);Purdue University(普渡大学)

Dezhong Zhao, Ruiqi Wang, Dayoon Suh, Taehyeon Kim, Ziqin Yuan, Byung-Cheol Min, Guohua Chen

AI总结:

本文提出PrefMMT多模态Transformer网络,通过解耦状态与动作模态并分层建模模态内时间依赖和模态间交互,提升基于偏好的强化学习中人类偏好建模精度,并在D4RL和Meta-World基准上超越现有方法。

摘要:

基于偏好的强化学习(PbRL)在使机器人行为与人类偏好对齐方面展现出潜力,但其成功在很大程度上依赖于通过奖励模型对人类偏好进行准确建模。大多数方法在偏好建模(PM)中采用马尔可夫假设,这忽略了机器人行为轨迹中影响人类评估的时间依赖性。尽管近期工作利用序列建模通过学习顺序非马尔可夫奖励来缓解这一问题,但它们忽略了机器人轨迹的多模态性质,这些轨迹由两种不同模态的元素组成:状态和动作。因此,它们往往难以捕捉这些模态之间复杂的相互作用,而这种相互作用在很大程度上塑造了人类偏好。在本文中,我们提出了一种用于偏好建模的多模态序列建模方法,通过解耦状态和动作模态来实现。我们引入了一个多模态Transformer网络,名为PrefMMT,它分层地利用模态内的时间依赖性和模态间的状态-动作交互来捕捉复杂的偏好模式。我们证明,PrefMMT在D4RL基准的运动任务和Meta-World基准的操控任务上,一致地优于最先进的偏好建模基线方法。

英文摘要:

Preference-based reinforcement learning (PbRL) shows promise in aligning robot behaviors with human preferences, but its success depends heavily on the accurate modeling of human preferences through reward models. Most methods adopt Markovian assumptions for preference modeling (PM), which overlook the temporal dependencies within robot behavior trajectories that impact human evaluations. While recent works have utilized sequence modeling to mitigate this by learning sequential non-Markovian rewards, they ignore the multimodal nature of robot trajectories, which consist of elements from two distinctive modalities: state and action. As a result, they often struggle to capture the complex interplay between these modalities that significantly shapes human preferences. In this paper, we propose a multimodal sequence modeling approach for PM by disentangling state and action modalities. We introduce a multimodal transformer network, named PrefMMT, which hierarchically leverages intra-modal temporal dependencies and inter-modal state-action interactions to capture complex preference patterns. We demonstrate that PrefMMT consistently outperforms state-of-the-art PM baselines on locomotion tasks from the D4RL benchmark and manipulation tasks from the Meta-World benchmark.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑