PrefCLM:利用众包大型语言模型增强基于偏好的强化学习
PrefCLM: Enhancing Preference-based Reinforcement Learning with Crowdsourced Large Language Models
AI总结:
针对PbRL依赖脚本教师合成反馈、难以适配个性化人机交互偏好的问题,提出PrefCLM框架,用Dempster-Shafer理论融合众包LLM偏好并引入人在回路精化,实验与用户研究验证其性能和满意度优势。
摘要:
基于偏好的强化学习(PbRL)正成为一种通过人类比较反馈来教授机器人的有前景方法,从而避开复杂奖励工程的需要。然而,现有PbRL方法所需的大量反馈往往导致依赖由脚本教师生成的合成反馈。这种方法再次需要复杂的奖励工程,并且难以适应人机交互(HRI)场景中特有的细微偏好;在这些场景中,用户可能对同一任务有独特期望。为应对这些挑战,我们提出PrefCLM,这是一种在PbRL中利用众包大型语言模型(LLM)作为模拟教师的新颖框架。我们采用Dempster-Shafer理论在分数层面融合来自多个LLM智能体的个体偏好,高效利用其多样性和集体智能。我们还引入了人在回路流程,可基于用户交互反馈促进集体精化。跨多种通用RL任务的实验结果表明,与传统脚本教师相比,PrefCLM取得了具有竞争力的性能,并擅长促成更自然、更高效的行为。一项真实世界用户研究(N=10)进一步证明,其能够根据个体用户偏好定制机器人行为,显著提升HRI场景中的用户满意度。
英文摘要:
Preference-based reinforcement learning (PbRL) is emerging as a promising approach to teaching robots through human comparative feedback, sidestepping the need for complex reward engineering. However, the substantial volume of feedback required in existing PbRL methods often lead to reliance on synthetic feedback generated by scripted teachers. This approach necessitates intricate reward engineering again and struggles to adapt to the nuanced preferences particular to human-robot interaction (HRI) scenarios, where users may have unique expectations toward the same task. To address these challenges, we introduce PrefCLM, a novel framework that utilizes crowdsourced large language models (LLMs) as simulated teachers in PbRL. We utilize Dempster-Shafer Theory to fuse individual preferences from multiple LLM agents at the score level, efficiently leveraging their diversity and collective intelligence. We also introduce a human-in-the-loop pipeline that facilitates collective refinements based on user interactive feedback. Experimental results across various general RL tasks show that PrefCLM achieves competitive performance compared to traditional scripted teachers and excels in facilitating more more natural and efficient behaviors. A real-world user study (N=10) further demonstrates its capability to tailor robot behaviors to individual user preferences, significantly enhancing user satisfaction in HRI scenarios.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。