探索CTR模型的缩放定律以提升在线性能
Exploring Scaling Laws of CTR Model for Online Performance Improvement
AI总结：
本文受大语言模型缩放定律启发，提出SUAN高等级CTR模型并通过在线蒸馏得到轻量级LightSUAN，在保持推理时间不变的同时显著提升在线CTR和CPM。
摘要：
CTR模型在许多在线个性化服务中对于改善用户体验和提升商业收入起着至关重要的作用。然而，当前的CTR模型普遍面临性能提升的瓶颈。受大语言模型（LLMs）缩放定律现象的启发，我们提出了一种提升CTR预测的新范式：首先，构建一个精度可随模型等级和数据规模扩展的CTR模型，然后将该模型中蕴含的知识蒸馏到能够在线服务用户的轻量级模型中。为将其付诸实践，我们构建了一个名为SUAN（Stacked Unified Attention Network，堆叠统一注意力网络）的CTR模型。在SUAN中，我们提出了UAB作为行为序列编码器。单个UAB统一了对序列特征和非序列特征的建模，并从多个视角衡量每个用户行为特征的重要性。堆叠的UAB将配置提升到高等级，为性能提升铺平了道路。为了受益于高等级SUAN的高性能并避免其推理时间长的缺点，我们通过稀疏自注意力和并行推理策略对SUAN进行修改，形成LightSUAN，然后采用在线蒸馏，以高等级SUAN为教师模型来训练低等级LightSUAN。蒸馏后的LightSUAN具有优越的性能，同时与LightSUAN保持相同的推理时间，非常适合在线部署。实验结果表明，SUAN表现异常出色，并在模型等级和数据规模上保持了跨越三个数量级的缩放定律，且蒸馏后的LightSUAN优于配置高一个等级的SUAN。更重要的是，蒸馏后的LightSUAN已集成到一项在线服务中，在保持平均推理时间可接受的同时，将CTR提升了2.81%，CPM提升了1.69%。我们的源代码可在https://github.com/laiweijiang/SUAN获取。
英文摘要：
CTR models play a vital role in improving user experience and boosting business revenue in many online personalized services. However, current CTR models generally encounter bottlenecks in performance improvement. Inspired by the scaling law phenomenon of LLMs, we propose a new paradigm for improving CTR predictions: first, constructing a CTR model with accuracy scalable to the model grade and data size, and then distilling the knowledge implied in this model into its lightweight model that can serve online users. To put it into practice, we construct a CTR model named SUAN (Stacked Unified Attention Network). In SUAN, we propose the UAB as a behavior sequence encoder. A single UAB unifies the modeling of the sequential and non-sequential features and also measures the importance of each user behavior feature from multiple perspectives. Stacked UABs elevate the configuration to a high grade, paving the way for performance improvement. In order to benefit from the high performance of the high-grade SUAN and avoid the disadvantage of its long inference time, we modify the SUAN with sparse self-attention and parallel inference strategies to form LightSUAN, and then adopt online distillation to train the low-grade LightSUAN, taking a high-grade SUAN as a teacher. The distilled LightSUAN has superior performance but the same inference time as the LightSUAN, making it well-suited for online deployment. Experimental results show that SUAN performs exceptionally well and holds the scaling laws spanning three orders of magnitude in model grade and data size, and the distilled LightSUAN outperforms the SUAN configured with one grade higher. More importantly, the distilled LightSUAN has been integrated into an online service, increasing the CTR by 2.81% and CPM by 1.69% while keeping the average inference time acceptable. Our source code is available at https://github.com/laiweijiang/SUAN.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
