Currently, a variety of pipeline tools are available for use in data engineering. Data scientists can use these tools to resolve data wrangling issues associated with data and accomplish some data engineering tasks from data ingestion through data preparation to utilization as input for machine learning (ML). Some of these tools have essential built-in components or can be combined with other tools to perform desired data engineering operations. While some tools are wholly or partly commercial, several open-source tools are available to perform expert-level data engineering tasks. This survey examines the broad categories and examples of pipeline tools based on their design and data engineering intentions. These categories are Extract Transform Load/Extract Load Transform (ETL/ELT), pipelines for Data Integration, Ingestion, and Transformation, Data Pipeline Orchestration and Workflow Management, and Machine Learning Pipelines. The survey also provides a broad outline of the utilization with examples within these broad groups and finally, a discussion is presented with case studies indicating the usage of pipeline tools for data engineering. The studies present some first-user application experiences with sample data, some complexities of the applied pipeline, and a summary note of approaches to using these tools to prepare data for machine learning.
A Survey of Automatic Prompt Engineering: An Optimization Perspective
School of Computer Science and Technology, Tongji University(同济大学计算机科学与技术学院);East China Normal University(华东师范大学);Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University(同济大学上海智能自主系统研究院)
Tracing the Heart: An Evidence-Linked Pipeline for Heart-Failure Feature Engineering
Nimblemind(宁敏德(音译));Singapore University of Technology and Design(新加坡科技设计大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Florida International University(佛罗里达国际大学);University of California Los Angeles(加利福尼亚大学洛杉矶分校);Rutgers University Newark(罗格斯大学纽瓦克分校);Rutgers Institute for Health Health Care Policy and Aging Research(罗格斯大学健康、医疗保健政策与老龄化研究所);Robert Wood Johnson Medical School(罗伯特·伍德·约翰逊医学院);Rutgers Health(罗格斯医疗)
A Survey of AI Methods for Geometry Preparation and Mesh Generation in Engineering Simulation
Sandia National Laboratories(桑迪亚国家实验室);Old Dominion University(旧 Dominion 大学);Los Alamos National Laboratory(洛斯阿拉莫斯国家实验室);New York University / Stony Brook University(纽约大学 / 斯通布鲁克大学);CEA(法国原子能委员会);Dalian University of Technology(大连理工大学);Carnegie Mellon University(卡内基梅隆大学)
Engineering Signals of Human-AI Collaboration in the Agentic Coding Era: A Longitudinal Analysis of 33,228 Pull Requests from vLLM and SGLang with Implications for Biomedical AI Agents and Bioinformatics Pipeline Developmen