减少模型抖动：生产环境中语义解析器的稳定再训练
Reducing Model Jitter: Stable Re-training of Semantic Parsers in Production Environments
摘要：
重新训练现代深度学习系统可能会导致模型性能的变化，即使在使用相同的数据和超参数进行训练时，也只是使用不同的随机种子。我们称这种现象为模型抖动。这个问题在生产环境中往往会加剧，因为模型是在噪声数据上重新训练的。在这项工作中，我们解决了稳定的再培训问题，重点是会话语义分析器。我们首先通过引入模型一致性度量并显示数据集噪声和模型大小的变化来量化模型抖动问题。然后，我们证明了各种抖动减少技术，如集成和蒸馏的有效性。最后，我们讨论了这种技术之间的实际权衡，并表明，共蒸馏提供了一个甜蜜点的抖动减少语义解析系统，只有适度增加资源的使用。
英文摘要：
Retraining modern deep learning systems can lead to variations in model performance even when trained using the same data and hyper-parameters by simply using different random seeds. We call this phenomenon model jitter. This issue is often exacerbated in production settings, where models are retrained on noisy data. In this work we tackle the problem of stable retraining with a focus on conversational semantic parsers. We first quantify the model jitter problem by introducing the model agreement metric and showing the variation with dataset noise and model sizes. We then demonstrate the effectiveness of various jitter reduction techniques such as ensembling and distillation. Lastly, we discuss practical trade-offs between such techniques and show that co-distillation provides a sweet spot in terms of jitter reduction for semantic parsing systems with only a modest increase in resource usage.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
