AgentDS Technical Report: Benchmarking the Future of Human-AI Collaboration in Domain-Specific Data Science
School of Statistics, University of Minnesota(明尼苏达大学统计学系);AIScientists, Inc.(AIScientists公司);Data Science Institute, University of Chicago(芝加哥大学数据科学研究所);Carlson School of Management, University of Minnesota(明尼苏达大学卡尔森管理学院);Cisco Research(思科研究);Department of Electrical and Computer Engineering, University of Minnesota(明尼苏达大学电气与计算机工程系);Division of Computational Health Sciences, University of Minnesota(明尼苏达大学计算健康科学 division)
An Luo, Jin Du, Xun Xian, Robert Specht, Fangqiao Tian, Ganghua Wang, Xuan Bi, Charles Fleming, Ashish Kundu, Jayanth Srinivasa, Mingyi Hong, Rui Zhang, Tianxi … 展开作者
An Luo, Jin Du, Xun Xian, Robert Specht, Fangqiao Tian, Ganghua Wang, Xuan Bi, Charles Fleming, Ashish Kundu, Jayanth Srinivasa, Mingyi Hong, Rui Zhang, Tianxi Li, Galin Jones, Jie Ding
Data science plays a critical role in transforming complex data into actionable insights across numerous domains. Recent developments in large language models (LLMs) and artificial intelligence (AI) agents have significantly automated data science workflow. However, it remains unclear to what extent AI agents can match the performance of human experts on domain-specific data science tasks, and in which aspects human expertise continues to provide advantages. We introduce AgentDS, a benchmark and competition designed to evaluate both AI agents and human-AI collaboration performance in domain-specific data science. AgentDS consists of 17 challenges across six industries: commerce, food production, healthcare, insurance, manufacturing, and retail banking. We conducted an open competition involving 29 teams and 80 participants, enabling systematic comparison between human-AI collaborative approaches and AI-only baselines. Our results show that current AI agents struggle with domain-specific reasoning. AI-only baselines perform below the top quartile of competition participants, while the strongest solutions arise from human-AI collaboration. These findings challenge the narrative of complete automation by AI and underscore the enduring importance of human expertise in data science, while illuminating directions for the next generation of AI. Visit the AgentDS website here: https://agentds.org/ and open source datasets here: https://huggingface.co/datasets/lainmn/AgentDS .
MedAgentBoard: Benchmarking Multi-Agent Collaboration with Conventional Methods for Diverse Medical Tasks
National Engineering Research Center for Software Engineering, Peking University(北京大学国家软件工程研究中心);School of Computing and Data Science, The University of Hong Kong(香港大学计算与数据科学学院);Centre for Medical Informatics, The University of Edinburgh(爱丁堡大学医学信息学中心);ETH Zurich(苏黎世联邦理工学院);Health Data Research UK(英国健康数据研究所);Key Laboratory of High Confidence Software Technologies, Ministry of Education(教育部高可信软件技术重点实验室)
CollabBench: Benchmarking and Unleashing Collaborative Ability of LLMs with Diverse Players via Proactive Engagement
Shanghai Institute of AI for Education(上海人工智能教育研究院);School of Computer Science(计算机科学学院);East China Normal University(东华大学);Tencent Inc.(腾讯公司);Shanghai Innovation Institute(上海创新研究院)
QuantBench: Benchmarking AI Methods for Quantitative Investment
The Hong Kong University of Science and Technology(香港科学与技术大学);IDEA Research(IDEA研究院);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));Beijing Institute of Technology(北京理工大学)
Collider-Bench: Benchmarking AI Agents with Particle Physics Analysis Reproduction
New High Energy Theory Center(新高能理论中心);Department of Physics & Astronomy(物理与天文学系);Rutgers University(罗格斯大学);Faculty of Computing & Data Sciences(计算与数据科学学院)
Position: Graph Learning Will Lose Relevance Due To Poor Benchmarks
Tel-Aviv University(特拉维夫大学);Meta;Technion - Israel Institute of Technology(以色列理工学院);University of Oxford(牛津大学);RWTH Aachen University(亚琛工业大学);NEC Laboratories Europe(欧洲NEC实验室);University of Stuttgart(斯图加特大学);Google Research(谷歌研究院)
Share, Collaborate, Benchmark: Advancing Travel Demand Research through rigorous open-source collaboration
University of California, Berkeley(加州大学伯克利分校);Global Metropolitan Studies(全球都市研究中心);Department of Civil and Environmental Engineering(土木与环境工程系);Department of City and Regional Planning(城市与区域规划系)
Advancing Human-Machine Teaming: Concepts, Challenges, and Applications
Virginia Tech(弗吉尼亚理工大学);C. G. Woodson High School(C. G. Woodson高中);US DEVCOM Army Research Laboratory(美国国防高级研究计划局陆军实验室);Korea Institute of Energy Technology (KENTECH)(韩国能源技术研究所);The University of Queensland(昆士兰大学)