arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.08335cs.LGcs.AIcs.DBstat.CO
收藏

数据工程流水线工具综述

A Survey of Pipeline Tools for Data Engineering

University of Aberdeen(阿伯丁大学)

Anthony Mbata, Yaji Sripada, Mingjun Zhong

AI总结:

本文是数据工程流水线工具综述,按设计与用途划分ETL/ELT、数据集成等四大类工具,结合实例介绍应用方式,并通过案例展示使用体验与为机器学习准备数据的方法。

摘要:

当前,数据工程领域已有多种流水线工具可供使用。数据科学家可借助这些工具解决与数据相关的数据整理问题,完成从数据接入、数据准备到作为机器学习(ML)输入使用的一系列数据工程任务。部分工具具备核心内置组件,或可与其他工具结合以执行所需的数据工程操作。其中一些工具为全部或部分商业化工具,也有若干开源工具可执行专业级数据工程任务。本综述基于工具的设计理念与数据工程用途,梳理了流水线工具的主要类别及实例,包括抽取-转换-加载/抽取-加载-转换(ETL/ELT)、数据集成、接入与转换流水线、数据流水线编排与工作流管理,以及机器学习流水线。综述还概述了各类工具的应用方式并附实例,最后通过案例研究讨论了流水线工具在数据工程中的应用,涵盖首批用户使用样本数据的应用体验、所应用流水线的部分复杂度,以及使用这些工具为机器学习准备数据的方法总结。

英文摘要:

Currently, a variety of pipeline tools are available for use in data engineering. Data scientists can use these tools to resolve data wrangling issues associated with data and accomplish some data engineering tasks from data ingestion through data preparation to utilization as input for machine learning (ML). Some of these tools have essential built-in components or can be combined with other tools to perform desired data engineering operations. While some tools are wholly or partly commercial, several open-source tools are available to perform expert-level data engineering tasks. This survey examines the broad categories and examples of pipeline tools based on their design and data engineering intentions. These categories are Extract Transform Load/Extract Load Transform (ETL/ELT), pipelines for Data Integration, Ingestion, and Transformation, Data Pipeline Orchestration and Workflow Management, and Machine Learning Pipelines. The survey also provides a broad outline of the utilization with examples within these broad groups and finally, a discussion is presented with case studies indicating the usage of pipeline tools for data engineering. The studies present some first-user application experiences with sample data, some complexities of the applied pipeline, and a summary note of approaches to using these tools to prepare data for machine learning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑