DataPrep.EDA:Python中以任务为中心的统计建模探索性数据分析
DataPrep.EDA: Task-Centric Exploratory Data Analysis for Statistical Modeling in Python
摘要:
探索性数据分析(EDA)是任何数据科学项目的关键步骤。然而,现有的Python库无法支持数据科学家完成统计建模的常见EDA任务。他们的API设计要么太低,这是优化绘图,而不是EDA,或太高的水平,这是很难指定更细粒度的EDA任务。作为回应,我们提出了DataPrep.EDA,一个新的以任务为中心的EDA系统在Python中。DataPrep.EDA允许数据科学家通过单个函数调用以声明方式指定不同粒度的各种EDA任务。我们指出了实现DataPrep.EDA所面临的一些挑战,并提出了有效的解决方案,以提高系统的可扩展性、可用性和可定制性。特别是,我们讨论了使用Dask构建EDA任务的数据处理管道的一些经验教训,并描述了我们加速管道的方法。我们进行了大量的实验,比较DataPrep.EDA与Pandas-profiling,Python中最先进的EDA系统。实验表明,DataPrep.EDA在速度和用户体验方面都明显优于Pandas-profiling。EDA作为DataPrep的EDA组件开放源代码:https://github.com/sfu-db/dataprep。
英文摘要:
Exploratory Data Analysis (EDA) is a crucial step in any data science project. However, existing Python libraries fall short in supporting data scientists to complete common EDA tasks for statistical modeling. Their API design is either too low level, which is optimized for plotting rather than EDA, or too high level, which is hard to specify more fine-grained EDA tasks. In response, we propose DataPrep.EDA, a novel task-centric EDA system in Python. DataPrep.EDA allows data scientists to declaratively specify a wide range of EDA tasks in different granularity with a single function call. We identify a number of challenges to implement DataPrep.EDA, and propose effective solutions to improve the scalability, usability, customizability of the system. In particular, we discuss some lessons learned from using Dask to build the data processing pipelines for EDA tasks and describe our approaches to accelerate the pipelines. We conduct extensive experiments to compare DataPrep.EDA with Pandas-profiling, the state-of-the-art EDA system in Python. The experiments show that DataPrep.EDA significantly outperforms Pandas-profiling in terms of both speed and user experience. DataPrep.EDA is open-sourced as an EDA component of DataPrep: https://github.com/sfu-db/dataprep.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。