arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.17845cs.HC
收藏

一种通过细粒度过程日志量化数据科学工作流的系统及试点研究

A System for Quantifying Data Science Workflows with Fine-Grained Procedural Logging and a Pilot Study

Jinjin Zhao, Avidgor Gal, Sanjay Krishnan

AI总结:

本文提出DataInquirer系统,通过追踪Jupyter笔记本中的增量代码执行来量化数据科学工作流,试点研究显示不同数据科学家分析同一数据时结论存在显著差异,且AI工具能使学生工作流更接近专家。

摘要:

对于研究人员而言,精确理解数据科学家如何将原始数据转化为洞察(包括典型的编程模式、工作流和方法论)至关重要。本文提出了一种名为DataInquirer的新系统,用于追踪Jupyter笔记本(一种计算笔记本)中的增量代码执行。该系统使我们能够定量测量数据科学任务中的时间、工作流和操作频率,而无需依赖人工标注或访谈。在一系列试点研究中,我们收集了97条轨迹,记录了四项研究中数据科学家的活动。虽然本文介绍了一个通用系统和数据分析方法,但我们在试点研究中聚焦于一个基础性子问题:不同数据科学家在分析相同数据时的一致性如何?我们将同一数据集上不同数据科学家之间的差异按三个类别进行归类:语义差异、句法差异和方法论差异。我们的结果表明,不同数据科学家在同一任务上得出的结论存在统计学上的显著差异,并为这一现象提供了定量证据。此外,我们的结果表明,AI驱动的代码工具会微妙地影响这些结果,使学生参与者能够生成更接近专家数据从业者的工作流。

英文摘要:

It is important for researchers to understand precisely how data scientists turn raw data into insights, including typical programming patterns, workflow, and methodology. This paper contributes a novel system, called DataInquirer, that tracks incremental code executions in Jupyter notebooks (a type of computational notebook). The system allows us to quantitatively measure timing, workflow, and operation frequency in data science tasks without resorting to human annotation or interview. In a series of pilot studies, we collect 97 traces, logging data scientist activities across four studies. While this paper presents a general system and data analysis approach, we focus on a foundational sub-question in our pilot studies: How consistent are different data scientists in analyzing the same data? We taxonomize variation between data scientists on the same dataset according to three categories: semantic, syntactic, and methodological. Our results suggest that there are statistically significant differences in the conclusions reached by different data scientists on the same task and present quantitative evidence for this phenomenon. Furthermore, our results suggest that AI-powered code tools subtly influence these results, allowing student participants to generate workflows that more resemble expert data practitioners.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑