三思而后行!设计以人为本的AI系统进行变更风险评估
Look Before You Leap! Designing a Human-Centered AI System for Change Risk Assessment
摘要:
减少生产系统中的故障数量是技术驱动的行业(例如在线零售行业)中最具挑战性的问题之一。为了应对这一挑战,变革管理已成为业务中一个很有前途的子领域,它以系统的方式管理和审查将在生产中部署的变革。但是,每天手动审查大量更改并评估与之相关的风险实际上是不可能的。这就需要开发一个自动化系统,以评估与大量变更相关的风险。有一些商业解决方案可用于解决这个问题,但这些解决方案缺乏将领域知识和领域专家的持续反馈纳入风险评估过程的能力。作为这项工作的一部分,我们的目标是通过在风险评估过程中建立一个连续的反馈循环,弥合模型驱动的变更请求风险评估和领域专家评估之间的差距。在这里,我们提出了我们的工作,以建立一个端到端的机器学习系统,以及我们所面临的一些实际挑战的讨论有关的极端偏态类分布,概念漂移,估计与模型的预测和系统的整体可扩展性的不确定性。
英文摘要:
Reducing the number of failures in a production system is one of the most challenging problems in technology driven industries, such as, the online retail industry. To address this challenge, change management has emerged as a promising sub-field in operations that manages and reviews the changes to be deployed in production in a systematic manner. However, it is practically impossible to manually review a large number of changes on a daily basis and assess the risk associated with them. This warrants the development of an automated system to assess the risk associated with a large number of changes. There are a few commercial solutions available to address this problem but those solutions lack the ability to incorporate domain knowledge and continuous feedback from domain experts into the risk assessment process. As part of this work, we aim to bridge the gap between model-driven risk assessment of change requests and the assessment of domain experts by building a continuous feedback loop into the risk assessment process. Here we present our work to build an end-to-end machine learning system along with the discussion of some of practical challenges we faced related to extreme skewness in class distribution, concept drift, estimation of the uncertainty associated with the model's prediction and the overall scalability of the system.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。