arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1902.08638cs.LGstat.ML
收藏

MPP:模型性能预测器

MPP: Model Performance Predictor

ParallelM(ParallelM(平行M)公司)

Sindhu Ghanta, Sriram Subramanian, Lior Khermosh, Harshil Shah, Yakov Goldberg, Swaminathan Sundararaman, Drew Roselli, Nisha Talagala

摘要:

操作是机器学习管道部署领域的一个关键挑战,涉及实时预测质量的监控和管理。通常,像准确性、RMSE等指标,用于跟踪部署中模型的性能。但是,由于没有标签,这些指标无法在生产中计算。我们建议使用ML算法,模型性能预测器(MPP),来跟踪部署中模型的性能。我们认为,这样的指标的合奏可以用来创建一个分数,代表在生产中的预测质量。这反过来又促进了ML警报的制定和定制,可以由运营团队升级到数据科学团队。这样的分数可以自动监控并实现大规模的ML部署。

英文摘要:

Operations is a key challenge in the domain of machine learning pipeline deployments involving monitoring and management of real-time prediction quality. Typically, metrics like accuracy, RMSE etc., are used to track the performance of models in deployment. However, these metrics cannot be calculated in production due to the absence of labels. We propose using an ML algorithm, Model Performance Predictor (MPP), to track the performance of the models in deployment. We argue that an ensemble of such metrics can be used to create a score representing the prediction quality in production. This in turn facilitates formulation and customization of ML alerts, that can be escalated by an operations team to the data science team. Such a score automates monitoring and enables ML deployments at scale.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑