arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.18488cs.AIcs.SE
收藏

AndroidControl-Curated:通过基准纯化揭示GUI药物的真正潜力

AndroidControl-Curated: Revealing the True Potential of GUI Agents through Benchmark Purification

BMW ArcherMind Information Technology Co. Ltd. (BA TechWorks)(宝马阿彻Mind信息技术有限公司(BA TechWorks))

Ho Fai Leung, Xiaoyan Xi, Fei Zuo

摘要:

像Siri和Google Assistant这样的设备上虚拟助手越来越重要,但它们的功能却因依赖于严格的、依赖于开发人员的API而受到限制。GUI代理提供了一个强大的,独立于API的替代方案,但它们的采用受到性能不佳的影响,因为即使是最好的模型(例如Qwen 3-VL-235 B)在AndroidControl等基准测试中的得分也限制在60%左右,远远不能用于现实世界的使用。我们的研究表明,问题不仅在于模型,而且在于基准本身。我们发现了AndroidControl中值得注意的缺点,包括模糊性和事实错误,这系统地低估了代理的能力。为了解决这个关键的疏忽,我们将AndroidControl增强为AndroidControl-Curated,这是通过严格的净化管道改进的基准测试的改进版本。在这个增强的基准测试中,最先进的模型在复杂任务上的成功率接近75%(提高了15%),这反映出设备上的GUI代理实际上比以前认为的更接近实际部署。我们介绍了我们的新SOTA模型Magma-R1- 3B,使用60小时的H20 GPU(约60美元)对2.4k精选样本进行后训练。尽管参数小了200倍,但该型号的性能可与Qwen 3- VL-235 B相媲美。我们向研究社区发布了AndroidControl-Curated基准测试和Magma-R1模型,鼓励采用这一增强的基准测试,以更好地反映模型功能,并加速开发强大的设备上虚拟助手。

英文摘要:

On-device virtual assistants like Siri and Google Assistant are increasingly pivotal, yet their capabilities are hamstrung by a reliance on rigid, developer-dependent APIs. GUI agents offer a powerful, API-independent alternative, but their adoption is hindered by the perception of poor performance, as even the best models (e.g. Qwen3-VL-235B) scores are capped at around 60% on benchmarks like AndroidControl, far from viability for real-world use. Our research reveals that issue lies not only with the models but with the benchmarks themselves. We identified notable shortcomings in AndroidControl, including ambiguities and factual errors, which systematically underrates agent capabilities. To address this critical oversight, we enhanced AndroidControl into AndroidControl-Curated, a refined version of the benchmark improved through a rigorous purification pipeline. On this enhanced benchmark, state-of-the-art models achieve success rates nearing 75% on complex tasks (15% improvement), reflecting that on-device GUI agents are actually closer to practical deployment than previously thought. We introduce our new SOTA model, Magma-R1- 3B, post-trained on just 2.4k curated samples using 60 hours of an H20 GPU (approximately $60). Despite being 200 times smaller in parameters, this model delivers performance comparable to Qwen3- VL-235B. We release both AndroidControl-Curated benchmark and Magma-R1 model to the research community, encouraging adoption of this enhanced benchmark to better reflect model capabilities and accelerate the development of robust, on-device virtual assistants.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑