arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.09856cs.CVcs.AIcs.CLcs.HC
收藏

Code2World: 通过可渲染代码生成实现一个GUI世界模型

Code2World: A GUI World Model via Renderable Code Generation

University of Science(科学大学);AMAP, Alibaba Group(AMAP,阿里巴巴集团);University of Oxford(牛津大学);Sun Yat-sen University(中山大学)

Yuhao Zheng, Li'an Zhong, Yi Wang, Rui Dai, Kaikui Liu, Xiangxiang Chu, Linyuan Lv, Philip Torr, Kevin Qinghong Lin

AI总结:

Code2World通过可渲染代码生成实现高保真UI预测,提升Android导航性能

摘要:

自主GUI代理通过感知界面并执行动作与环境互动。作为虚拟沙盒,GUI世界模型通过使能动作条件预测赋予代理人类般的前瞻性。然而,现有基于文本和像素的方法难以同时实现高视觉保真度和细粒度结构可控性。为此,我们提出Code2World,一种视觉-语言编码器,通过可渲染代码生成模拟下一个视觉状态。具体而言,为解决数据稀缺问题,我们构建AndroidCode,将GUI轨迹转化为高保真的HTML并通过视觉反馈修订机制优化合成代码,得到超过80000对高质量屏幕-动作对。为适应现有VLMs进行代码预测,我们首先进行SFT作为冷启动以遵循格式布局,然后进一步应用Render-Aware Reinforcement Learning,通过强制视觉语义保真度和动作一致性使用渲染结果作为奖励信号。大量实验表明,Code2World-8B在下一个UI预测中表现最佳,与竞争性的GPT-5和Gemini-3-Pro-Image相当。值得注意的是,Code2World以灵活的方式显著提高了在AndroidWorld导航中的下游导航成功率,将Gemini-2.5-Flash提升9.5%。代码可在https://github.com/AMAP-ML/Code2World获取。

英文摘要:

Autonomous GUI agents interact with environments by perceiving interfaces and executing actions. As a virtual sandbox, the GUI World model empowers agents with human-like foresight by enabling action-conditioned prediction. However, existing text- and pixel-based approaches struggle to simultaneously achieve high visual fidelity and fine-grained structural controllability. To this end, we propose Code2World, a vision-language coder that simulates the next visual state via renderable code generation. Specifically, to address the data scarcity problem, we construct AndroidCode by translating GUI trajectories into high-fidelity HTML and refining synthesized code through a visual-feedback revision mechanism, yielding a corpus of over 80K high-quality screen-action pairs. To adapt existing VLMs into code prediction, we first perform SFT as a cold start for format layout following, then further apply Render-Aware Reinforcement Learning which uses rendered outcome as the reward signal by enforcing visual semantic fidelity and action consistency. Extensive experiments demonstrate that Code2World-8B achieves the top-performing next UI prediction, rivaling the competitive GPT-5 and Gemini-3-Pro-Image. Notably, Code2World significantly enhances downstream navigation success rates in a flexible manner, boosting Gemini-2.5-Flash by +9.5% on AndroidWorld navigation. The code is available at https://github.com/AMAP-ML/Code2World.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑