arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.22701cs.AI
收藏

Best-of-Q: 通过推理中的Q函数动作排名提升VLM代理

Best-of-Q: Improving VLM agents with Q-function Action Ranking at Inference

Emilien Biré, María Santos, Kai Yuan

AI总结:

Best-of-Q通过在推理过程中利用Q函数动作排名提升VLM代理性能,显著提高任务成功率。

摘要:

视觉-语言模型(VLMs)已成为代理在数字环境如网络和操作系统中自主操作的强大骨干。然而,这些模型在快速变化的环境中如网络中存在不适应性,这可以通过微调来缓解,但需要大量的模型训练和数据收集。在本工作中,我们引入了一种新的范式,用于在推理过程中增强代理的VLM策略,而无需策略再训练。从根本上说,我们的方法将VLM作为高容量动作生成器的角色与最终动作选择机制分离。我们保持VLM策略冻结,并使用它为给定状态生成一组候选动作。然后,一个轻量级、离线训练的Q函数对这些候选者进行重新排序,代理执行估计价值最高的动作。主要贡献是将Q函数直接应用于推理过程以实现即时的策略改进,而不是离线用于重新标记数据以进行策略再训练。我们在学术WebVoyager基准上展示了我们的方法显著提高了代理的成功率,将Qwen2.5-VL-7B代理的成功率从38.8%提高到55.7%,将专有GPT-4.1代理的成功率从82.4%提高到88.8%。

英文摘要:

Vision-Language Models (VLMs) have become powerful backbones for agents to autonomously operate in digital environments like the web and operating systems. However, these models suffer from inadaptability to fast-changing environments like the web, which can be alleviated by fine-tuning requiring expansive model training and data collection. In this work, we introduce a novel paradigm for enhancing agentic VLM policies at inference without policy retraining. Fundamentally, our approach decouples the VLM's role as a high-capacity action proposer from the final action selection mechanism. We keep the VLM policy frozen and use it to generate a set of candidate actions for a given state. Then, a lightweight, offline-trained Q-function reranks these candidates, and the agent executes the action with the highest estimated value. The main contribution is to apply the Q-function directly during inference for immediate policy improvement, and not offline to relabel data for policy retraining. We demonstrate on the academic WebVoyager benchmark that our method significantly boosts agent success rates, improving a Qwen2.5-VL-7B agent from 38.8% to 55.7% and a proprietary GPT-4.1 agent from 82.4% to 88.8%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑