在端到端数据科学项目上评估AI性能
Benchmarking AI Performance on End-to-End Data Science Projects
AI总结:
本研究评估了AI模型在端到端数据科学项目中的表现,发现不同模型在结构化任务和判断任务上的表现差异显著,表明AI在常规任务上可模拟初级数据科学家,但需进一步验证。
摘要:
数据科学是一种整合技术、分析、沟通和伦理技能的综合工作流程,但当前的AI基准测试主要关注组成部分。我们测试AI模型能否生成端到端的数据科学项目。为此,我们创建了一个包含40个端到端数据科学项目及其关联评分标准的基准测试。我们利用这些测试用例构建了一个自动评分管道,系统地评估生成式AI模型生成的数据科学项目。我们发现,生成式AI模型在完成端到端数据科学项目方面的表现因模型而异。最近的模型在结构化任务上表现良好,但在需要判断的任务上存在显著差异。这些发现表明,尽管AI模型可以在常规任务上近似初级数据科学家,但仍然需要验证。
英文摘要:
Data science is an integrated workflow of technical, analytical, communication, and ethical skills, but current AI benchmarks focus mostly on constituent parts. We test whether AI models can generate end-to-end data science projects. To do this we create a benchmark of 40 end-to-end data science projects with associated rubric evaluations. We use these to build an automated grading pipeline that systematically evaluates the data science projects produced by generative AI models. We find the extent to which generative AI models can complete end-to-end data science projects varies considerably by model. Most recent models did well on structured tasks, but there were considerable differences on tasks that needed judgment. These findings suggest that while AI models could approximate entry-level data scientists on routine tasks, they require verification.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。