COVR：视觉控制中视觉语言模型与强化学习代理的协同优化
COVR:Collaborative Optimization of VLMs and RL Agent for Visual-Based Control
AI总结：
COVR通过协同优化视觉语言模型与强化学习代理，利用RL生成数据提升VLM语义推理能力，并通过动作先验引导策略学习，实现视觉控制任务中的高效优化。
摘要：
视觉强化学习（RL）在复杂任务中由于高维观测而导致样本效率低下。尽管现有研究表明视觉语言模型（VLMs）可以辅助RL，但它们往往侧重于从VLM向RL的知识蒸馏，忽略了RL生成的交互数据增强VLM的潜力。为此，我们提出了COVR，一种协同优化框架，使VLM和RL策略相互增强。具体而言，COVR通过RL生成的数据微调VLM，以增强与目标任务一致的语义推理能力，并利用增强的VLM进一步通过动作先验引导策略学习。为提高微调效率，我们引入了两个关键模块：（1）探索驱动动态过滤模块，通过自适应阈值保留有价值的探索样本；（2）回报感知自适应损失权重模块，通过RL的回报信号量化采样动作的一致性，以提高训练稳定性。我们进一步设计了一种渐进微调策略以减少资源消耗。大量实验表明，COVR在各种具有挑战性的视觉控制任务中均取得了优异的性能。
英文摘要：
Visual reinforcement learning (RL) suffers from poor sample efficiency due to high-dimensional observations in complex tasks. While existing works have shown that vision-language models (VLMs) can assist RL, they often focus on knowledge distillation from the VLM to RL, overlooking the potential of RL-generated interaction data to enhance the VLM. To address this, we propose COVR, a collaborative optimization framework that enables the mutual enhancement of the VLM and RL policies. Specifically, COVR fine-tunes the VLM with RL-generated data to enhance the semantic reasoning ability consistent with the target task, and uses the enhanced VLM to further guide policy learning via action priors. To improve fine-tuning efficiency, we introduce two key modules: (1) an Exploration-Driven Dynamic Filter module that preserves valuable exploration samples using adaptive thresholds based on the degree of exploration, and (2) a Return-Aware Adaptive Loss Weight module that improves the stability of training by quantifying the inconsistency of sampling actions via return signals of RL. We further design a progressive fine-tuning strategy to reduce resource consumption. Extensive experiments show that COVR achieves strong performance across various challenging visual control tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
