arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.19005cs.LGcs.AIstat.ME
收藏

AgentDS技术报告:领域特定数据科学中人机协作的未来基准测试

AgentDS Technical Report: Benchmarking the Future of Human-AI Collaboration in Domain-Specific Data Science

School of Statistics, University of Minnesota(明尼苏达大学统计学系);AIScientists, Inc.(AIScientists公司);Data Science Institute, University of Chicago(芝加哥大学数据科学研究所);Carlson School of Management, University of Minnesota(明尼苏达大学卡尔森管理学院);Cisco Research(思科研究);Department of Electrical and Computer Engineering, University of Minnesota(明尼苏达大学电气与计算机工程系);Division of Computational Health Sciences, University of Minnesota(明尼苏达大学计算健康科学 division)
An Luo, Jin Du, Xun Xian, Robert Specht, Fangqiao Tian, Ganghua Wang, Xuan Bi, Charles Fleming, Ashish Kundu, Jayanth Srinivasa, Mingyi Hong, Rui Zhang, Tianxi … 展开作者

An Luo, Jin Du, Xun Xian, Robert Specht, Fangqiao Tian, Ganghua Wang, Xuan Bi, Charles Fleming, Ashish Kundu, Jayanth Srinivasa, Mingyi Hong, Rui Zhang, Tianxi Li, Galin Jones, Jie Ding

AI总结:

提出AgentDS基准测试和竞赛,通过17个跨行业挑战评估AI代理及人机协作在领域特定数据科学中的表现,发现AI代理在领域推理上存在不足,人机协作优于纯AI方法。

摘要:

数据科学在将复杂数据转化为跨领域的可操作洞察方面发挥着关键作用。大型语言模型(LLM)和人工智能(AI)代理的最新发展显著自动化了数据科学工作流程。然而,目前尚不清楚AI代理在多大程度上能够匹配人类专家在领域特定数据科学任务上的表现,以及人类专业知识在哪些方面仍具有优势。我们引入了AgentDS,一个旨在评估AI代理和人机协作在领域特定数据科学中表现的基准测试和竞赛。AgentDS包含来自六个行业(商业、食品生产、医疗保健、保险、制造业和零售银行)的17个挑战。我们组织了一场公开竞赛,涉及29支队伍和80名参与者,从而能够系统比较人机协作方法与纯AI基线。我们的结果表明,当前的AI代理在领域特定推理方面存在困难。纯AI基线的表现低于竞赛参与者的前四分位数,而最强的解决方案来自人机协作。这些发现挑战了AI完全自动化的说法,并强调了人类专业知识在数据科学中的持久重要性,同时为下一代AI指明了方向。访问AgentDS网站:https://agentds.org/,开源数据集:https://huggingface.co/datasets/lainmn/AgentDS。

英文摘要:

Data science plays a critical role in transforming complex data into actionable insights across numerous domains. Recent developments in large language models (LLMs) and artificial intelligence (AI) agents have significantly automated data science workflow. However, it remains unclear to what extent AI agents can match the performance of human experts on domain-specific data science tasks, and in which aspects human expertise continues to provide advantages. We introduce AgentDS, a benchmark and competition designed to evaluate both AI agents and human-AI collaboration performance in domain-specific data science. AgentDS consists of 17 challenges across six industries: commerce, food production, healthcare, insurance, manufacturing, and retail banking. We conducted an open competition involving 29 teams and 80 participants, enabling systematic comparison between human-AI collaborative approaches and AI-only baselines. Our results show that current AI agents struggle with domain-specific reasoning. AI-only baselines perform below the top quartile of competition participants, while the strongest solutions arise from human-AI collaboration. These findings challenge the narrative of complete automation by AI and underscore the enduring importance of human expertise in data science, while illuminating directions for the next generation of AI. Visit the AgentDS website here: https://agentds.org/ and open source datasets here: https://huggingface.co/datasets/lainmn/AgentDS .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑