arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.21568cs.SE
收藏

从零散脚本到协同流程:构建一个用于开发者生产力指标的鲁棒ELT框架

From Ad-Hoc Scripts to Orchestrated Pipelines: Architecting a Resilient ELT Framework for Developer Productivity Metrics

Adobe Inc.(Adobe公司)

Yuvraj Agrawal, Pallav Jain

AI总结:

本文提出通过DAG编排和Medallion架构构建鲁棒ELT流程,以提升开发者生产力指标的可靠性与可持续性。

摘要:

开发者生产力仪表盘对于可视化DevOps性能指标如部署频率和变更失败率(DORA)至关重要。然而,这些仪表板的效用经常受到数据可靠性问题的阻碍。在我们平台的早期版本中,零散的数据摄入脚本(Cron任务)导致了"沉默失败",即数据缺口在数天内未被发现,损害了组织信任。本文报告了我们从传统调度迁移到使用有向无环图(DAG)编排和Medallion架构的稳健提取-加载-转换(ELT)流程的经验。我们详细说明了将数据提取与转换解耦的操作优势,不可变的原始历史记录对于指标重定义的必要性,以及状态基于的依赖管理的实现。我们的经验表明,将指标流程视为生产级分布式系统是可持续工程分析的前提条件。

英文摘要:

Developer Productivity Dashboards are essential for visualizing DevOps performance metrics such as Deployment Frequency and Change Failure Rate (DORA). However, the utility of these dashboards is frequently undermined by data reliability issues. In early iterations of our platform, ad-hoc ingestion scripts (Cron jobs) led to "silent failures," where data gaps went undetected for days, eroding organizational trust. This paper reports on our experience migrating from legacy scheduling to a robust Extract-Load-Transform (ELT) pipeline using Directed Acyclic Graph (DAG) orchestration and Medallion Architecture. We detail the operational benefits of decoupling data extraction from transformation, the necessity of immutable raw history for metric redefinition, and the implementation of state-based dependency management. Our experience suggests that treating the metrics pipeline as a production-grade distributed system is a prerequisite for sustainable engineering analytics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑