Computer use agents automate digital tasks by directly interacting with graphical user interfaces (GUIs) on computers and mobile devices, offering significant potential to enhance human productivity by completing an open-ended space of user queries. However, current agents face significant challenges: imprecise grounding of GUI elements, difficulties with long-horizon task planning, and performance bottlenecks from relying on single generalist models for diverse cognitive tasks. To this end, we introduce Agent S2, a novel compositional framework that delegates cognitive responsibilities across various generalist and specialist models. We propose a novel Mixture-of-Grounding technique to achieve precise GUI localization and introduce Proactive Hierarchical Planning, dynamically refining action plans at multiple temporal scales in response to evolving observations. Evaluations demonstrate that Agent S2 establishes new state-of-the-art (SOTA) performance on three prominent computer use benchmarks. Specifically, Agent S2 achieves 18.9% and 32.7% relative improvements over leading baseline agents such as Claude Computer Use and UI-TARS on the OSWorld 15-step and 50-step evaluation. Moreover, Agent S2 generalizes effectively to other operating systems and applications, surpassing previous best methods by 52.8% on WindowsAgentArena and by 16.52% on AndroidWorld relatively. Code available at https://github.com/simular-ai/Agent-S.
Dexterous manipulation of arbitrary objects, a fundamental daily task for humans, has been a grand challenge for autonomous robotic systems. Although data-driven approaches using reinforcement learning can develop specialist policies that discover behaviors to control a single object, they often exhibit poor generalization to unseen ones. In this work, we show that policies learned by existing reinforcement learning algorithms can in fact be generalist when combined with multi-task learning and a well-chosen object representation. We show that a single generalist policy can perform in-hand man…
Generalist-Specialist Mixture-of-Experts for Rare Pathology Detection in Multimodal Imaging
Technical University of Munich (TUM)（慕尼黑工业大学）；TUM University Hospital（慕尼黑工业大学附属医院）；Hasso Plattner Institute for Digital Engineering（哈索·普拉特纳数字工程研究所）；University of Potsdam（波茨坦大学）；Imperial College London（伦敦帝国理工学院）
From Specialist to Generalist: Unlocking SAM's Learning Potential on Unlabeled Medical Images
Carnegie Mellon University（卡内基梅隆大学）；Industrial University of Ho Chi Minh City（胡志明市工业大学）；University of North Carolina at Charlotte（北卡罗来纳州夏洛特大学）；University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）
TAGS: A Test-Time Generalist-Specialist Framework with Retrieval-Augmented Reasoning and Verification
Mohamed bin Zayed University of Artificial Intelligence（莫莫德·本·扎耶德人工智能大学）；Monash University（墨尔本大学）；Xi’an Jiaotong-Liverpool University（西安交通大学利物浦大学）；University of Southampton（南安普顿大学）
From Generalist to Specialist: A Context-Fusion Framework for Endoscopic Polyp Reporting with a Frozen VLM
Zhejiang University（浙江大学）；Shanghai Institute for Advanced Study, Zhejiang University（浙江大学上海高等研究院）；Shanghai Key Laboratory of MICCAI（上海市MICCAI重点实验室）；Digital Medical Research Center, School of Basic Medical Sciences, Fudan University（复旦大学基础医学院数字医学研究中心）；Endoscopy Center and Endoscopy Research Institute, Zhongshan Hospital, Fudan University（复旦大学附属中山医院内镜中心及内镜研究所）；Shanghai Collaborative Innovation Center of Endoscopy（上海市内镜协同创新中心）；Alliance Manchester Business School, The University of Manchester（曼彻斯特大学联盟曼彻斯特商学院）
DeepMind（深度思维）；University of Oxford（牛津大学）；IDSIA, USI Lugano（卢加诺瑞士意大利语大学人工智能研究所）；Mila, Université de Montréal（蒙特利尔大学米拉人工智能实验室）；Purdue University（普渡大学）
The cornerstone of neural algorithmic reasoning is the ability to solve algorithmic tasks, especially in a way that generalises out of distribution. While recent years have seen a surge in methodological improvements in this area, they mostly focused on building specialist models. Specialist models are capable of learning to neurally execute either only one algorithm or a collection of algorithms with identical control-flow backbone. Here, instead, we focus on constructing a generalist neural algorithmic learner -- a single graph neural network processor capable of learning to execute a wide r…
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of California, San Diego（加州大学圣地亚哥分校）；The Hong Kong Polytechnic University（香港理工大学）；University of Michigan（密歇根大学）；Nanyang Technological University（南洋理工大学）；Johns Hopkins University（约翰霍普金斯大学）；University of Tübingen（图宾根大学）
