操作化机器学习:一项访谈研究
Operationalizing Machine Learning: An Interview Study
摘要:
组织依赖于机器学习工程师(MLE)来操作ML,即,在生产中部署和维护ML管道。ML(或MLOps)的操作过程包括以下几个方面的持续循环:(i)数据汇集和标记,(ii)实验以提高ML性能,(iii)在多阶段部署过程中进行评估,以及(iv)监控生产中的性能下降。当综合考虑时,这些责任看起来令人震惊--任何人都是如何做MLOp的,有哪些未解决的挑战,以及对工具构建者的影响是什么?我们对18位MLE进行了半结构化的人种学采访,这些MLE在许多应用程序中工作,包括聊天机器人、自动驾驶汽车和金融。我们的访谈揭示了决定生产ML部署成功的三个变量:速度,验证和版本控制。我们总结了成功的ML实验,部署和维持生产性能的常见实践。最后,我们讨论了受访者的痛点和反模式,以及对工具设计的影响。
英文摘要:
Organizations rely on machine learning engineers (MLEs) to operationalize ML, i.e., deploy and maintain ML pipelines in production. The process of operationalizing ML, or MLOps, consists of a continual loop of (i) data collection and labeling, (ii) experimentation to improve ML performance, (iii) evaluation throughout a multi-staged deployment process, and (iv) monitoring of performance drops in production. When considered together, these responsibilities seem staggering -- how does anyone do MLOps, what are the unaddressed challenges, and what are the implications for tool builders? We conducted semi-structured ethnographic interviews with 18 MLEs working across many applications, including chatbots, autonomous vehicles, and finance. Our interviews expose three variables that govern success for a production ML deployment: Velocity, Validation, and Versioning. We summarize common practices for successful ML experimentation, deployment, and sustaining production performance. Finally, we discuss interviewees' pain points and anti-patterns, with implications for tool design.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。