arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.11896cs.LG
收藏

DigiRL:用自主强化学习训练真实环境设备控制智能体

DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning

UC Berkeley(加州大学伯克利分校);UIUC(伊利诺伊大学厄巴纳-香槟分校);Google DeepMind(谷歌DeepMind)

Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, Aviral Kumar

AI总结:

针对视觉语言模型缺乏决策数据、真实设备控制表现不佳的问题,提出两阶段自主强化学习方法DigiRL,在AitW数据集上大幅超越现有方法,刷新真实环境设备控制数字智能体的最优性能。

摘要:

视觉语言模型(VLM)的训练语料通常缺乏足够的以决策为中心的数据,这使得现成的VLM在通过图形用户界面(GUI)进行真实环境设备控制等决策任务上表现欠佳。虽然使用静态演示的训练方法已展现出一定前景,但我们发现这类方法在控制真实GUI时存在不足,因为它们无法应对静态观测数据未涵盖的现实世界随机性和非平稳性。\n本文提出了一种名为DigiRL的新型自主强化学习方法,用于训练真实环境设备控制智能体,该方法通过两个阶段微调预训练VLM:首先用离线强化学习初始化模型,随后进行离线到在线的强化学习。为此,我们构建了一个可扩展、可并行的Android学习环境,配备了基于VLM的评估器,并开发了一种简单却有效的强化学习方法以在该领域进行学习。我们的方法采用优势加权强化学习,其优势估计器经过增强以处理随机性,同时还配备了自动课程学习机制以获取最大的学习信号。\n我们使用Android-in-the-Wild(AitW)数据集验证了DigiRL的有效性:我们用强化学习训练的13亿参数VLM,相较于使用静态人类演示数据的监督微调,成功率从17.7%提升至67.2%,绝对提升幅度达49.5%。这些结果不仅大幅超越了此前的最优智能体,包括搭载GPT-4V的AppAgent(成功率8.3%)和用AitW数据训练的170亿参数CogAgent(38.5%),还超越了此前基于过滤行为克隆的最优自主强化学习方法(57.8%),从而为真实环境设备控制的数字智能体确立了新的 state-of-the-art。

英文摘要:

Training corpuses for vision language models (VLMs) typically lack sufficient amounts of decision-centric data. This renders off-the-shelf VLMs sub-optimal for decision-making tasks such as in-the-wild device control through graphical user interfaces (GUIs). While training with static demonstrations has shown some promise, we show that such methods fall short for controlling real GUIs due to their failure to deal with real-world stochasticity and non-stationarity not captured in static observational data. This paper introduces a novel autonomous RL approach, called DigiRL, for training in-the-wild device control agents through fine-tuning a pre-trained VLM in two stages: offline RL to initialize the model, followed by offline-to-online RL. To do this, we build a scalable and parallelizable Android learning environment equipped with a VLM-based evaluator and develop a simple yet effective RL approach for learning in this domain. Our approach runs advantage-weighted RL with advantage estimators enhanced to account for stochasticity along with an automatic curriculum for deriving maximal learning signal. We demonstrate the effectiveness of DigiRL using the Android-in-the-Wild (AitW) dataset, where our 1.3B VLM trained with RL achieves a 49.5% absolute improvement -- from 17.7 to 67.2% success rate -- over supervised fine-tuning with static human demonstration data. These results significantly surpass not only the prior best agents, including AppAgent with GPT-4V (8.3% success rate) and the 17B CogAgent trained with AitW data (38.5%), but also the prior best autonomous RL approach based on filtered behavior cloning (57.8%), thereby establishing a new state-of-the-art for digital agents for in-the-wild device control.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑