自由形式偏好学习用于机器人操作
Freeform Preference Learning for Robotic Manipulation
AI总结：
提出自由形式偏好学习（FPL），通过自然语言定义偏好轴并学习语言条件奖励模型，在长时域操作任务中比稀疏奖励和二元偏好方法提升38个百分点，支持行为组合与测试时策略引导。
摘要：
奖励设计仍然是自主机器人策略改进的核心瓶颈，尤其是在长时域操作任务中，稀疏的成功标签提供的信号太少，而二元偏好将许多竞争性的质量概念压缩成一个模糊的信号。我们提出自由形式偏好学习（FPL），一种从自由形式人类偏好中学习机器人策略的方法。FPL 不是让标注者判断两个轨迹哪个整体更好，而是让它们定义自然语言的偏好轴，例如速度、安全性、放置质量或细心程度，并沿着每个轴提供成对偏好。这些注释用于学习一个语言条件奖励模型，该模型将轨迹和偏好标签映射到特定轴的奖励。我们使用该模型训练一个奖励条件策略，该策略在多个人类指定的维度上进行优化。在四个真实世界和两个模拟的长时域操作任务中，FPL 比稀疏奖励和二元偏好方法提高了38个百分点。除了性能提升外，FPL 无需显式子任务分割即可学习密集的进度信号，展现出数据中不存在的行为组合性，并允许用户在测试时无需重新训练即可引导策略朝向不同行为。博客文章及视频见此 https URL。
英文摘要：
Reward design remains a central bottleneck for autonomous robot policy improvement, especially in long-horizon manipulation tasks where sparse success labels provide too little signal and binary preferences collapse many competing notions of quality into one ambiguous signal. We introduce Freeform Preference Learning (FPL), a method for learning robot policies from freeform human preferences. Rather than asking annotators which of two trajectories is better overall, FPL lets them define natural-language preference axes, such as speed, safety, quality of placement, or carefulness, and provide pairwise preferences along each axis. These annotations are used to learn a language-conditioned reward model that maps a trajectory and preference label to an axis-specific reward. We use this model to train a reward-conditioned policy that optimizes across the multiple human-specified dimensions. Across four real-world and two simulated long-horizon manipulation tasks, FPL improves over sparse-reward and binary-preference methods by 38 percentage points. Beyond improved performance, FPL learns dense progress signals without explicit subtask segmentation, shows compositionality of behavior not present in the data, and allows users to steer the policy towards different behaviors at test time without retraining. Blog post with videos available at https://freeform-pl.github.io/fpl.website/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
