CoAdapt-GUI：面向未知GUI应用的工作流上下文与策略联合适配
CoAdapt-GUI: Joint Workflow Context and Policy Adaptation for Unseen GUI Applications
浏览论文内容
中文总结 AI 辅助
CoAdapt-GUI框架通过联合适配工作流上下文与策略，在有限交互预算无目标演示的未知GUI应用场景下，将AndroidWorld泛化性能提至45.0%、AndroidWorld Plus提至52.9%，优于仅策略TTA基线。
中文摘要 AI 辅助
将移动GUI智能体部署到源训练中未包含的应用时，其性能仍存在脆弱性。本文研究在有限目标交互预算且无目标演示的条件下的新应用泛化问题。我们提出CoAdapt-GUI，这是一种测试时适配（TTA）框架，可基于智能体自身的目标应用滚动交互与奖励，联合适配结构化工作流上下文和策略。工作流上下文保留可迁移的流程、失败模式及验证规则，同时排除与特定应用绑定的源细节。这种分离机制使可复用的工作流知识能在不迁移源界面状态的情况下指导适配。对于策略适配，与任务上下文匹配的组相对优化会在冻结的视觉语言模型上更新LoRA适配器。在两项未知应用评估中，CoAdapt-GUI在AndroidWorld泛化任务上达到45.0%的性能，而已报道的仅策略TTA基线为37.5%；同时将AndroidWorld Plus的性能从38.6%提升至52.9%。这些结果表明，受迁移约束的工作流上下文能带来显著性能提升，而联合策略适配可进一步改善保留的性能。
英文摘要
Mobile GUI agents remain brittle when deployed to applications absent from source training. We study novel-app generalization under a limited target interaction budget and without target demonstrations. We introduce CoAdapt-GUI, a test-time adaptation (TTA) framework that jointly adapts structured workflow context and policy from the agent's own target-app rollouts and rewards. The workflow context retains transferable procedures, failure modes, and verification rules while excluding app-bound source details. This separation allows reusable workflow knowledge to guide adaptation without transferring source-interface state. For policy adaptation, task-context-matched group-relative optimization updates a LoRA adapter on a frozen vision-language model. Across two unseen-app evaluations, CoAdapt-GUI reaches 45.0% on AndroidWorld-Generalization, compared with 37.5% for the reported Policy-Only TTA baseline, and raises AndroidWorld Plus performance from 38.6% to 52.9%. These results show that transfer-constrained workflow context provides substantial gains and that joint policy adaptation further improves held-out performance.
发表机构
- Li Gu（李谷（音译））
机构由 AI 辅助整理，请以论文原文为准。
