arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.05333cs.SE
收藏

ML赋能系统的模型部署与监控:现状与问题

ML-Enabled Systems Model Deployment and Monitoring: Status Quo and Problems

Eduardo Zimelewicz, Marcos Kalinowski, Daniel Mendez, Görkem Giray, Antonio Pedro Santos Alves, Niklas Lavesson, Kelly Azevedo, Hugo Villamizar, Tatiana Escoved… 展开作者

Eduardo Zimelewicz, Marcos Kalinowski, Daniel Mendez, Görkem Giray, Antonio Pedro Santos Alves, Niklas Lavesson, Kelly Azevedo, Hugo Villamizar, Tatiana Escovedo, Helio Lopes, Stefan Biffl, Juergen Musil, Michael Felderer, Stefan Wagner, Teresa Baldassarre, Tony Gorschek

AI总结:

本研究通过国际调查(188份回复)揭示ML模型部署与监控的现状:部署常为独立服务且MLOps采用有限,监控缺失且指标选择困难,为问题驱动的后续研究提供指导。

摘要:

[背景] 集成机器学习(ML)模型的系统,通常称为ML赋能系统,已变得司空见惯。然而,关于ML赋能系统在实践中如何被工程化的经验证据仍然有限,尤其是围绕ML模型传播的活动。[目标] 我们调查了与ML模型传播相关的当代工业实践和问题,重点关注模型部署和ML生命周期阶段的监控。[方法] 我们进行了一项国际调查,以收集从业者对ML赋能系统如何工程化的见解。我们共收集了来自25个国家的188份完整回复。我们分析了模型部署和监控阶段报告的现状与问题。我们使用自助法(bootstrapping)及置信区间分析当代实践,并对报告的问题应用开放编码和轴向编码程序进行了定性分析。[结果] 从业者认为模型部署和监控阶段既相关又困难。关于模型部署,模型通常作为独立服务部署,MLOps原则的采用有限。报告的问题包括生产部署基础设施架构设计的困难以及遗留应用集成。关于模型监控,许多生产中的模型未被监控。主要监控的方面是输入、输出和决策。报告的问题涉及缺乏监控实践、需要创建自定义监控工具以及选择合适指标。[结论] 我们的结果有助于更好地理解实践中的采用实践和问题,并支持以问题驱动的方式指导ML部署和监控研究。

英文摘要:

[Context] Systems incorporating Machine Learning (ML) models, often called ML-enabled systems, have become commonplace. However, empirical evidence on how ML-enabled systems are engineered in practice is still limited, especially for activities surrounding ML model dissemination. [Goal] We investigate contemporary industrial practices and problems related to ML model dissemination, focusing on the model deployment and the monitoring of ML life cycle phases. [Method] We conducted an international survey to gather practitioner insights on how ML-enabled systems are engineered. We gathered a total of 188 complete responses from 25 countries. We analyze the status quo and problems reported for the model deployment and monitoring phases. We analyzed contemporary practices using bootstrapping with confidence intervals and conducted qualitative analyses on the reported problems applying open and axial coding procedures. [Results] Practitioners perceive the model deployment and monitoring phases as relevant and difficult. With respect to model deployment, models are typically deployed as separate services, with limited adoption of MLOps principles. Reported problems include difficulties in designing the architecture of the infrastructure for production deployment and legacy application integration. Concerning model monitoring, many models in production are not monitored. The main monitored aspects are inputs, outputs, and decisions. Reported problems involve the absence of monitoring practices, the need to create custom monitoring tools, and the selection of suitable metrics. [Conclusion] Our results help provide a better understanding of the adopted practices and problems in practice and support guiding ML deployment and monitoring research in a problem-driven manner.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑