PREDILECT：强化学习中基于零样本语言推理的偏好刻画
PREDILECT: Preferences Delineated with Zero-Shot Language-based Reasoning in Reinforcement Learning
AI总结：
PREDILECT通过将人类文本提示与偏好结合，利用LLM零样本推理提取高信息量状态-动作对，重构奖励学习目标，从而提升基于偏好的强化学习样本效率，并在模拟社交导航中验证了训练策略的有效性。
摘要：
基于偏好的强化学习（RL）已成为机器人学习中的一个新兴领域，人类通过表达对不同状态-动作对序列的偏好，在塑造机器人行为方面发挥着关键作用。然而，为机器人制定现实可行的策略需要人类对大量查询作出响应。在这项工作中，我们通过扩展每次查询所收集的信息，使其同时包含偏好和可选的文本提示，来应对样本效率挑战。为实现这一目标，我们利用大型语言模型（LLM）的零样本能力，从人类提供的文本中进行推理。为了适应额外的查询信息，我们重新构建了奖励学习目标，使其包含灵活的突出部分——即包含相对较高信息量、且与从预训练LLM以零样本方式处理的特征相关的状态-动作对。在模拟场景和用户研究中，我们通过分析反馈及其影响，揭示了本工作的有效性。此外，所收集的集体反馈用于在模拟社交导航环境中训练机器人沿符合社会规范的轨迹行动。我们在 https://sites.google.com/view/rl-predilect 提供了训练策略的视频示例。
英文摘要：
Preference-based reinforcement learning (RL) has emerged as a new field in robot learning, where humans play a pivotal role in shaping robot behavior by expressing preferences on different sequences of state-action pairs. However, formulating realistic policies for robots demands responses from humans to an extensive array of queries. In this work, we approach the sample-efficiency challenge by expanding the information collected per query to contain both preferences and optional text prompting. To accomplish this, we leverage the zero-shot capabilities of a large language model (LLM) to reason from the text provided by humans. To accommodate the additional query information, we reformulate the reward learning objectives to contain flexible highlights -- state-action pairs that contain relatively high information and are related to the features processed in a zero-shot fashion from a pretrained LLM. In both a simulated scenario and a user study, we reveal the effectiveness of our work by analyzing the feedback and its implications. Additionally, the collective feedback collected serves to train a robot on socially compliant trajectories in a simulated social navigation landscape. We provide video examples of the trained policies at https://sites.google.com/view/rl-predilect
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
