Large language models (LLMs) have empowered intelligent agents to execute intricate tasks within domain-specific software such as browsers and games. However, when applied to general-purpose software systems like operating systems, LLM agents face three primary challenges. Firstly, the action space is vast and dynamic, posing difficulties for LLM agents to maintain an up-to-date understanding and deliver accurate responses. Secondly, real-world tasks often require inter-application cooperation}, demanding farsighted planning from LLM agents. Thirdly, agents need to identify optimal solutions aligning with user constraints, such as security concerns and preferences. These challenges motivate AndroidArena, an environment and benchmark designed to evaluate LLM agents on a modern operating system. To address high-cost of manpower, we design a scalable and semi-automated method to construct the benchmark. In the task evaluation, AndroidArena incorporates accurate and adaptive metrics to address the issue of non-unique solutions. Our findings reveal that even state-of-the-art LLM agents struggle in cross-APP scenarios and adhering to specific constraints. Additionally, we identify a lack of four key capabilities, i.e., understanding, reasoning, exploration, and reflection, as primary reasons for the failure of LLM agents. Furthermore, we provide empirical analysis on the failure of reflection, and improve the success rate by 27% with our proposed exploration strategy. This work is the first to present valuable insights in understanding fine-grained weakness of LLM agents, and offers a path forward for future research in this area. Environment, benchmark, and evaluation code for AndroidArena are released at https://github.com/AndroidArenaAgent/AndroidArena.
See-Control: A Multimodal Agent Framework for Smartphone Interaction with a Robotic Arm
University College London（伦敦大学学院）；Imperial College London（伦敦帝国学院）；Huawei Noah’s Ark Lab（华为诺亚实验室）；ShanghaiTech University（上海科技大学）；Southern University of Science（南方科技大学）
PowerLens: Taming LLM Agents for Safe and Personalized Mobile Power Management
China University of Geosciences (Beijing)（中国地质大学（北京））；Shenzhen University（深圳大学）；Hong Kong University of Science and Technology（香港科学与技术大学）；City University of Hong Kong（香港城市大学）
OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesis
Nanjing University（南京大学）；SenseTime（秒速）；Nanyang Technological University（南洋理工大学）；Shanghai AI Laboratory（上海人工智能实验室）；The University of Hong Kong（香港大学）；Xi’an Jiaotong University（西安交通大学）
Hi-Agent: Hierarchical Vision-Language Agents for Mobile Device Control
Tsinghua University（清华大学）；Huawei Noah’s Ark Lab（华为诺亚实验室）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University College London（伦敦大学学院）
Nanyang Technological University（南洋理工大学）；MiLM Plus, Xiaomi Inc.（小米公司）；Independent Researchers（独立研究人员）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）；Wuhan University（武汉大学）；Xiamen University（厦门大学）
CyberForge: Verified Vulnerability Injection at Repository Level for Cybersecurity Agent Training
National Institute of Standards and Technology（美国国家标准与技术研究院）；Université Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；CNRS（法国国家科学研究中心）；University of Maryland, College Park（马里兰大学帕克分校）
Mind the Gap: Action Rebinding Attacks against Android GUI Agents
State Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；Hornor Device Co., Ltd（Hornor设备有限公司）；Institute of Dataspace, Hefei Comprehensive National Science Center（数据空间研究所，合肥综合性国家科学中心）
Benchmarking LLM Judges for Mobile Agent Evaluation
Mila – Québec AI Institute（米拉-魁北克人工智能研究所）；Concordia University（康考迪亚大学）；University of Toronto（多伦多大学）；Shanghai University（上海大学）；McMaster University（麦克马斯特大学）
PhoneWorld: From Real-App Trajectories to Dynamic and Verifiable Environments for Phone-Use Agents
Tencent Hunyuan（腾讯文英）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院（ Gallagher 学院））
