MobileDreamer: 用于GUI代理的生成式草图世界模型
MobileDreamer: Generative Sketch World Model for GUI Agent
AI总结：
MobileDreamer通过生成式草图世界模型和rollout想象策略，提升GUI代理在长周期任务中的决策能力，任务成功率提升5.25%。
摘要：
移动GUI代理在现实世界自动化和实际应用中展现出强大潜力。然而，大多数现有代理仍然是反应式的，决策主要基于当前屏幕，这限制了它们在长周期任务上的性能。通过构建基于重复交互的世界模型，可以预测动作结果并支持移动GUI代理更好的决策。这具有挑战性，因为模型必须在保持高效性的同时，能够预测动作后的状态并具有空间意识。在本文中，我们提出了MobileDreamer，一种高效的基于世界模型的前瞻框架，用于为基于世界模型的GUI代理提供未来想象。它由文本草图世界模型和GUI代理的rollout想象组成。文本草图世界模型通过学习过程将数字图像转换为关键任务相关的草图，以预测动作后的状态，并设计了一种新的顺序不变学习策略以保留GUI元素的空间信息。GUI代理的rollout想象策略通过利用世界模型的预测能力优化动作选择过程。在Android World上的实验表明，MobileDreamer实现了最先进的性能，任务成功率提高了5.25%。世界模型评估进一步验证了我们的文本草图建模能够准确预测关键的GUI元素。
英文摘要：
Mobile GUI agents have shown strong potential in real-world automation and practical applications. However, most existing agents remain reactive, making decisions mainly from current screen, which limits their performance on long-horizon tasks. Building a world model from repeated interactions enables forecasting action outcomes and supports better decision making for mobile GUI agents. This is challenging because the model must predict post-action states with spatial awareness while remaining efficient enough for practical deployment. In this paper, we propose MobileDreamer, an efficient world-model-based lookahead framework to equip the GUI agents based on the future imagination provided by the world model. It consists of textual sketch world model and rollout imagination for GUI agent. Textual sketch world model forecasts post-action states through a learning process to transform digital images into key task-related sketches, and designs a novel order-invariant learning strategy to preserve the spatial information of GUI elements. The rollout imagination strategy for GUI agent optimizes the action-selection process by leveraging the prediction capability of world model. Experiments on Android World show that MobileDreamer achieves state-of-the-art performance and improves task success by 5.25%. World model evaluations further verify that our textual sketch modeling accurately forecasts key GUI elements.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
