Autonomous agents that accomplish complex computer tasks with minimal human interventions have the potential to transform human-computer interaction, significantly enhancing accessibility and productivity. However, existing benchmarks either lack an interactive environment or are limited to environments specific to certain applications or domains, failing to reflect the diverse and complex nature of real-world computer use, thereby limiting the scope of tasks and agent scalability. To address this issue, we introduce OSWorld, the first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across various operating systems such as Ubuntu, Windows, and macOS. OSWorld can serve as a unified, integrated computer environment for assessing open-ended computer tasks that involve arbitrary applications. Building upon OSWorld, we create a benchmark of 369 computer tasks involving real web and desktop apps in open domains, OS file I/O, and workflows spanning multiple applications. Each task example is derived from real-world computer use cases and includes a detailed initial state setup configuration and a custom execution-based evaluation script for reliable, reproducible evaluation. Extensive evaluation of state-of-the-art LLM/VLM-based agents on OSWorld reveals significant deficiencies in their ability to serve as computer assistants. While humans can accomplish over 72.36% of the tasks, the best model achieves only 12.24% success, primarily struggling with GUI grounding and operational knowledge. Comprehensive analysis using OSWorld provides valuable insights for developing multimodal generalist agents that were not possible with previous benchmarks. Our code, environment, baseline models, and data are publicly available at https://os-world.github.io.
When Stories Evolve: Benchmarking LLM Storytelling Across Agent Architectures in Open-Ended World Simulations
The University of Hong Kong(香港大学);Peking University(北京大学);Tsinghua University(清华大学);Beijing Institute of Mathematical Sciences and Applications (BIMSA)(北京数学科学与应用研究院)
StarDojo: Benchmarking Open-Ended Behaviors of Agentic Multimodal LLMs in Production-Living Simulations with Stardew Valley
Nanyang Technological University(南洋理工大学);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));East China University of Science and Technology(华东理工大学);Singapore Management University(新加坡管理大学)
OSWorld-Science: A Benchmark of Computer Use Agents for Learning and Using Scientific Software
Tsinghua University(清华大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);RIKEN AIP(理化学研究所人工智能中心);University of Waterloo(滑铁卢大学);Carnegie Mellon University(卡内基梅隆大学);Yale University(耶鲁大学);Northwestern University(西北大学);Zhejiang University(浙江大学);University of California, Berkeley(加利福尼亚大学伯克利分校);University of Illinois Chicago(伊利诺伊大学芝加哥分校);Boston University(波士顿大学);The University of Hong Kong(香港大学);Stanford University(斯坦福大学);New York University(纽约大学);The University of Tokyo(东京大学)
Automated Creativity Evaluation of Language Models Across Open-Ended Tasks
Raffles Institution(莱佛士书院);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院);Lee Kong Chian School of Medicine, Nanyang Technological University(南洋理工大学李光前医学院);Centre of AI in Medicine (C-AIM), Nanyang Technological University(南洋理工大学人工智能医学中心)
Benchmarking Large Multimodal Models for Ophthalmic Visual Question Answering with OphthalWeChat
The Hong Kong Polytechnic University(香港理工大学);Swiss Federal Institute of Technology Lausanne (EPFL)(洛桑联邦理工学院);PolyU-Wuhan Technology and Innovation Research Institute(香港理工大学武汉技术创新研究院);University of Liverpool(利物浦大学);Centre for Eye and Vision Research (CEVR)(眼科视觉研究中心)
MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors
Department of Computer Science, ETH Zurich(苏黎世联邦理工学院计算机科学系);ETH AI Center(ETH人工智能中心);Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany(达姆施塔特技术大学计算机科学系);Professorship for Learning Sciences and Higher Education, ETH Zurich(苏黎世联邦理工学院学习科学与高等教育教授职位)
AnthroDial: Benchmarking LLM Anthropomorphism in Autonomous Social Interaction
Shanghai Institute of Innovation(上海创新研究院);University of Science and Technology of China(中国科学技术大学);East China Normal University(华东师范大学);Anhui University(安徽大学);Shanghai Jiaotong University(上海交通大学);Fudan University(复旦大学);University of Melbourne(墨尔本大学);Zhejiang University(浙江大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Shanghai Tianyou Software Co., Ltd.(上海天游软件有限公司);Chabiyue (Shanghai) Information Technology Co., Ltd.(茶百悦(上海)信息技术有限公司);Zhejiang Century Huatong Group Co., Ltd.(浙江世纪华通集团股份有限公司)
D2VBench: Benchmarking Large Language Models with Value Dilemmas in Daily Scenarios
TJUNLP Lab, School of Computer Science and Technology, Tianjin University(天津大学计算机科学与技术学院TJUNLP实验室);The International Joint Institute of Tianjin University(天津大学国际联合研究院)
HIVMedQA: Benchmarking large language models for HIV medical decision support
Department of Biosystems Science and Engineering, ETH Zurich, Basel, Switzerland(1 生物系统科学与工程系,苏黎世联邦理工学院,巴塞尔,瑞士);School of Life Sciences, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland(2 生命科学学院,洛桑联邦理工学院,洛桑,瑞士);Swiss Institute of Bioinformatics, Lausanne, Switzerland(3 瑞士生物信息学研究所,洛桑,瑞士);Biomedical Data Science Center, Lausanne University Hospital and University of Lausanne, Lausanne, Switzerland(4 生物医学数据科学中心,洛桑大学医院和洛桑大学,洛桑,瑞士);Institute of Medical Virology, University of Zurich, Zurich, Switzerland(5 医学病毒学研究所,苏黎世大学,苏黎世,瑞士);Department of Infectious Diseases and Hospital Epidemiology, University Hospital Zurich, Zurich, Switzerland(6 感染病与医院流行病学系,苏黎世大学医院,苏黎世,瑞士);ETH AI Center, ETH Zurich, Zurich, Switzerland(7 ETH人工智能中心,苏黎世联邦理工学院,苏黎世,瑞士);Department of Quantitative Biomedicine, University of Zurich, Zurich, Switzerland(8 量化生物医学系,苏黎世大学,苏黎世,瑞士)
Benchmarking Foundation Models with Language-Model-as-an-Examiner
Singapore Management University(新加坡管理大学);Tsinghua University(清华大学);University of California, Los Angeles(加州大学洛杉矶分校);Beijing University of Posts and Telecommunications(北京邮电大学)