arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.02654cs.LGcs.AIstat.ML
收藏

通过蒸馏减少客户流失

Churn Reduction via Distillation

Google Research(谷歌研究院)

Heinrich Jiang, Harikrishna Narasimhan, Dara Bahri, Andrew Cotter, Afshin Rostamizadeh

摘要:

在现实世界的系统中,随着更多的数据变得可用,模型会频繁更新,除了实现高准确性之外,目标还在于保持与基础模型相比的预测差异较小(即预测性“流失”)。如果模型重新训练导致行为大不相同,那么它可能会对下游系统造成负面影响,特别是如果这种扰动可以避免,对模型准确性的影响有限。在本文中,我们展示了使用基本模型作为教师的蒸馏训练与对预测流失的显式约束的训练之间的等价性。然后,我们证明了蒸馏在低流失训练中的表现非常出色,与广泛的数据集和模型架构(包括全连接网络,卷积网络和Transformers)的一些最新基线相比。

英文摘要:

In real-world systems, models are frequently updated as more data becomes available, and in addition to achieving high accuracy, the goal is to also maintain a low difference in predictions compared to the base model (i.e. predictive "churn"). If model retraining results in vastly different behavior, then it could cause negative effects in downstream systems, especially if this churn can be avoided with limited impact on model accuracy. In this paper, we show an equivalence between training with distillation using the base model as the teacher and training with an explicit constraint on the predictive churn. We then show that distillation performs strongly for low churn training against a number of recent baselines on a wide range of datasets and model architectures, including fully-connected networks, convolutional networks, and transformers.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑