arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.08516cs.CV
收藏

MolmoWeb:开放视觉网络代理和开放数据用于开放网络

MolmoWeb: Open Visual Web Agent and Open Data for the Open Web

Allen Institute for AI(艾伦人工智能研究所);University of Washington(华盛顿大学)
Tanmay Gupta, Piper Wolters, Zixian Ma, Peter Sushko, Rock Yuren Pang, Diego Llanes, Yue Yang, Taira Anderson, Boyuan Zheng, Zhongzheng Ren, Harsh Trivedi, Tayl… 展开作者

Tanmay Gupta, Piper Wolters, Zixian Ma, Peter Sushko, Rock Yuren Pang, Diego Llanes, Yue Yang, Taira Anderson, Boyuan Zheng, Zhongzheng Ren, Harsh Trivedi, Taylor Blanton, Caleb Ouellette, Winson Han, Ali Farhadi, Ranjay Krishna

AI总结:

本文提出MolmoWeb,一个开放的多模态网络代理,通过公开的浏览器任务演示和网页GUI感知数据,实现状态-of-the-art的网络代理性能,促进开放研究。

摘要:

网络代理——能够代表用户在网页上导航和执行任务的自主系统——有潜力改变人们与数字世界互动的方式。然而,目前最强大的网络代理仍然依赖于专有模型,这些模型使用未公开的训练数据和配方,限制了科学理解、可重复性和社区驱动的进步。我们相信,为开放网络构建的代理应该在开放中构建。为此,我们介绍了(1)MolmoWebMix,一个包含浏览器任务演示和网页GUI感知数据的大而多样化的混合体,以及(2)MolmoWeb,一个完全开放的多模态网络代理家族。具体来说,MolmoWebMix结合了来自多个互补生成管线的超过10万条合成任务轨迹,以及30000多条人类演示、原子网页技能轨迹和GUI感知数据,包括指代表达接地和截图问答。MolmoWeb代理作为指令条件的视觉-语言动作策略运行:给定一个任务指令和网页截图,它们预测下一步的浏览器操作,无需访问HTML、可访问性树或专用API。在浏览器使用基准测试如WebVoyager、Online-Mind2Web和DeepShop上,MolmoWeb代理以状态-of-the-art的结果超越了类似规模的开放权重-only模型,如Fara-7B、UI-Tars-1.5-7B和Holo1-7B。MolmoWeb-8B还超越了基于更大封闭前沿模型如GPT-4o构建的集合标记(SoM)代理。我们进一步通过测试时缩放通过并行滚出与最佳-N选择实现了一致的增益,在WebVoyager和Online-Mind2Web上分别达到94.7%和60.5%的pass@4(与78.2%和35.3%的pass@1相比)。我们将发布模型检查点、训练数据、代码和统一的评估工具包,以促进可重复性和加速网络代理的开放研究。

英文摘要:

Web agents--autonomous systems that navigate and execute tasks on the web on behalf of users--have the potential to transform how people interact with the digital world. However, the most capable web agents today rely on proprietary models with undisclosed training data and recipes, limiting scientific understanding, reproducibility, and community-driven progress. We believe agents for the open web should be built in the open. To this end, we introduce (1) MolmoWebMix, a large and diverse mixture of browser task demonstrations and web-GUI perception data and (2) MolmoWeb, a family of fully open multimodal web agents. Specifically, MolmoWebMix combines over 100K synthetic task trajectories from multiple complementary generation pipelines with 30K+ human demonstrations, atomic web-skill trajectories, and GUI perception data, including referring expression grounding and screenshot question answering. MolmoWeb agents operate as instruction-conditioned visual-language action policies: given a task instruction and a webpage screenshot, they predict the next browser action, requiring no access to HTML, accessibility trees, or specialized APIs. Available in 4B and 8B size, on browser-use benchmarks like WebVoyager, Online-Mind2Web, and DeepShop, MolmoWeb agents achieve state-of-the-art results outperforming similar scale open-weight-only models such as Fara-7B, UI-Tars-1.5-7B, and Holo1-7B. MolmoWeb-8B also surpasses set-of-marks (SoM) agents built on much larger closed frontier models like GPT-4o. We further demonstrate consistent gains through test-time scaling via parallel rollouts with best-of-N selection, achieving 94.7% and 60.5% pass@4 (compared to 78.2% and 35.3% pass@1) on WebVoyager and Online-Mind2Web respectively. We will release model checkpoints, training data, code, and a unified evaluation harness to enable reproducibility and accelerate open research on web agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑