The advancement of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) has catalyzed the development of mobile graphic user interface (GUI) AI agents, which is designed to autonomously perform tasks on mobile devices. However, a significant gap persists in mobile GUI agent evaluation, where existing benchmarks predominantly rely on either static frame assessments such as AndroidControl or offline static apps such as AndroidWorld and thus fail to capture agent performance in dynamic, real-world online mobile apps. To address this gap, we present Android Agent Arena (A3), a novel "essential-state" based procedural evaluation system for mobile GUI agents. A3 introduces a benchmark of 100 tasks derived from 20 widely-used, dynamic online apps across 20 categories from the Google Play Store, ensuring evaluation comprehension. A3 also presents a novel "essential-state" based procedural evaluation method that leverages MLLMs as reward models to progressively verify task completion and process achievement. This evaluation approach address the limitations of traditional function based evaluation methods on online dynamic apps. Furthermore, A3 includes a toolkit to streamline Android device interaction, reset online environment and apps and facilitate data collection from both human and agent demonstrations. The complete A3 system, including the benchmark and tools, will be publicly released to provide a robust foundation for future research and development in mobile GUI agents.
STT-Arena: A More Realistic Environment for Tool-Using with Spatio-Temporal Dynamics
Hong Kong Institute of AI for Science, City University of Hong Kong（香港人工智能科学研究院，香港城市大学）；Department of Data Science, City University of Hong Kong（数据科学系，香港城市大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Li Auto Inc.（李汽有限公司）；Independent Researcher（独立研究者）
CV-Arena: An Open Benchmark for Instructional Computer Vision Problem Solving with Human-AI Collaborative Preferences
Texas A&M University（德克萨斯A&M大学）；Worcester Polytechnic Institute（沃斯特理工大学）；Tohoku University（东北大学）；Georgia Institute of Technology（佐治亚理工学院）；NVIDIA（英伟达）；UCSB（加州大学圣塔芭芭拉分校）；UC Merced（加州大学默塞德分校）
RecSys Arena: Pair-wise Recommender System Evaluation with Large Language Models
School of New Media and Communication, Tianjin University（天津大学新媒体与传播学院）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
Nanyang Technological University（南洋理工大学）；MiLM Plus, Xiaomi Inc.（小米公司）；Independent Researchers（独立研究人员）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）；Wuhan University（武汉大学）；Xiamen University（厦门大学）
GUI Testing Arena: A Unified Benchmark for Advancing Autonomous GUI Testing Agent
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Om AI Research（Om AI研究院）；Binjiang Institute of Zhejiang University（浙江大学滨江研究院）
Mind the Gap: Action Rebinding Attacks against Android GUI Agents
State Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；Hornor Device Co., Ltd（Hornor设备有限公司）；Institute of Dataspace, Hefei Comprehensive National Science Center（数据空间研究所，合肥综合性国家科学中心）
Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence
University of Oxford（牛津大学）；Lighthouse（灯塔公司）；Beihang University（北京航空航天大学）；Hebei University of Technology（河北工业大学）；Imperial College London（帝国理工学院）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
MobiFlow: Real-World Mobile Agent Benchmarking through Trajectory Fusion
Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University（上海交通大学并行与分布式系统研究所（IPADS））；Department of Precision Instrument, Tsinghua University（清华大学精密仪器系）；National Innovation Institute of High-end Smart Appliances（国家高端智能家电创新研究院）
GUIGuard-Bench: Toward a General Evaluation for Privacy-Preserving GUI Agents
Beijing Normal University（北京师范大学）；Zhongguancun Academy（中关村学院）；University of Science and Technology of China（中国科学技术大学）；A*STAR；Zhongguancun Institution of Artificial Intelligence（中关村人工智能研究所）
PhoneWorld: From Real-App Trajectories to Dynamic and Verifiable Environments for Phone-Use Agents
Tencent Hunyuan（腾讯文英）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院（ Gallagher 学院））
