离线 RLAIF：通过 SFO 引导 VLM 反馈用于强化学习
Offline RLAIF: Piloting VLM Feedback for RL via SFO
AI总结：
针对离线强化学习中如何整合 VLM 反馈的挑战，提出子轨迹过滤优化（SFO）框架，并设计子轨迹过滤行为克隆（SFBC）方法，通过回顾性过滤机制移除失败前的子轨迹，显著提升了基于视觉反馈的智能体学习的鲁棒性。
摘要：
尽管互联网规模的图像和文本数据在视觉语言模型（VLMs）中实现了强大的泛化能力，但缺乏同等规模的互联网控制数据阻碍了标准强化学习（RL）智能体实现类似的泛化。尽管 VLMs 由于缺乏动作条件化的训练数据，在解决控制任务方面存在根本限制，但它们的图像理解能力使其能够通过识别成功结果在 RL 任务中提供有价值的反馈。AI 反馈强化学习（RLAIF）的一个关键挑战是如何最好地将 VLM 衍生的信号整合到学习过程中。我们在离线 RL 背景下探索了这个问题，并引入了一类称为子轨迹过滤优化（SFO）的方法。我们确定了三个关键见解。首先，轨迹长度在离线 RL 中起着至关重要的作用，因为全轨迹偏好学习会加剧拼接问题，因此需要使用子轨迹。其次，即使在马尔可夫环境中，也需要来自图像序列的非马尔可夫奖励信号来评估轨迹的改进，因为 VLMs 不解释控制动作，必须依赖随时间推移的视觉线索。第三，一种简单而有效的方法——过滤和加权的行为克隆——始终优于更复杂的基于 RLHF 的方法。我们提出了子轨迹过滤行为克隆（SFBC），这是一种利用对子轨迹的 VLM 反馈的方法，同时结合了回顾性过滤机制，该机制移除失败之前的子轨迹以提高鲁棒性并防止动荡。请欣赏我们的机场双关语。
英文摘要：
While internet-scale image and textual data have enabled strong generalization in Vision-Language Models (VLMs), the absence of internet-scale control data has impeded the development of similar generalization in standard reinforcement learning (RL) agents. Although VLMs are fundamentally limited in their ability to solve control tasks due to their lack of action-conditioned training data, their capacity for image understanding allows them to provide valuable feedback in RL tasks by recognizing successful outcomes. A key challenge in Reinforcement Learning from AI Feedback (RLAIF) is determining how best to integrate VLM-derived signals into the learning process. We explore this question in the context of offline RL and introduce a class of methods called Sub-Trajectory Filtered Optimization (SFO). We identify three key insights. First, trajectory length plays a crucial role in offline RL, as full-trajectory preference learning exacerbates the stitching problem, necessitating the use of sub-trajectories. Second, even in Markovian environments, a non-Markovian reward signal from a sequence of images is required to assess trajectory improvement, as VLMs do not interpret control actions and must rely on visual cues over time. Third, a simple yet effective approach--filtered and weighted behavior cloning--consistently outperforms more complex RLHF-based methods. We propose Sub-Trajectory Filtered Behavior Cloning (SFBC), a method that leverages VLM feedback on sub-trajectories while incorporating a retrospective filtering mechanism that removes sub-trajectories preceding failures to improve robustness and prevent turbulence. Please enjoy our airport puns.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
