arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.07703cs.AIcs.CL
收藏

DSBench:数据科学智能体离成为数据科学专家还有多远?

DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?

University of Texas at Dallas(德克萨斯大学达拉斯分校);Tencent AI Lab(腾讯AI实验室);University of Southern California(南加州大学)

Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, Dong Yu

AI总结:

本文提出DSBench基准,通过466个数据分析任务和74个数据建模任务评估数据科学智能体,发现最佳智能体仅解决34.12%的分析任务,揭示了现有模型与真实专家水平之间的显著差距。

摘要:

大型语言模型(LLMs)和大型视觉语言模型(LVLMs)已经展现出令人印象深刻的语言/视觉推理能力,推动了近期构建面向特定应用(如购物助手或AI软件工程师)的智能体的趋势。最近,许多数据科学基准被提出,以考察它们在数据科学领域的表现。然而,与真实世界的数据科学应用相比,现有的数据科学基准由于设置过于简化,仍然存在不足。为了弥合这一差距,我们引入了DSBench,这是一个综合基准,旨在用现实任务评估数据科学智能体。该基准包含466个数据分析任务和74个数据建模任务,来源于Eloquence和Kaggle竞赛。DSBench通过涵盖长上下文、多模态任务背景、基于大文件和多表结构的推理,以及执行端到端数据建模任务,提供了现实的环境。我们对最先进的LLMs、LVLMs和智能体的评估表明,它们在大多数任务上表现挣扎,表现最好的智能体仅解决了34.12%的数据分析任务,并实现了34.74%的相对性能差距(RPG)。这些发现强调了在开发更实用、更智能、更自主的数据科学智能体方面需要进一步取得进展。

英文摘要:

Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) have demonstrated impressive language/vision reasoning abilities, igniting the recent trend of building agents for targeted applications such as shopping assistants or AI software engineers. Recently, many data science benchmarks have been proposed to investigate their performance in the data science domain. However, existing data science benchmarks still fall short when compared to real-world data science applications due to their simplified settings. To bridge this gap, we introduce DSBench, a comprehensive benchmark designed to evaluate data science agents with realistic tasks. This benchmark includes 466 data analysis tasks and 74 data modeling tasks, sourced from Eloquence and Kaggle competitions. DSBench offers a realistic setting by encompassing long contexts, multimodal task backgrounds, reasoning with large data files and multi-table structures, and performing end-to-end data modeling tasks. Our evaluation of state-of-the-art LLMs, LVLMs, and agents shows that they struggle with most tasks, with the best agent solving only 34.12% of data analysis tasks and achieving a 34.74% Relative Performance Gap (RPG). These findings underscore the need for further advancements in developing more practical, intelligent, and autonomous data science agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑