大规模推荐系统的分布式等效替代训练
Distributed Equivalent Substitution Training for Large-Scale Recommender Systems
摘要：
我们提出了分布式等效替换（DES）训练，一种新的分布式训练框架，用于大规模推荐系统的动态稀疏特征。DES通过减少通信，首次将完全同步的训练引入到大规模推荐系统中，从而使商业推荐系统的训练收敛更快，达到更好的CTR。DES通过用计算上等价的子运算符替换权重丰富的运算符并聚合部分结果而不是直接通过网络传输巨大的稀疏权重，从而需要更少的通信。由于在大规模深度学习推荐模型（DLRM）上使用同步训练，DES实现了更高的AUC（ROC下面积）。我们成功地将DES培训应用于工业场景的多个流行DLRM。实验表明，我们的实现优于国家的最先进的基于PS的训练框架，实现高达68.7%的通信节省和更高的吞吐量相比，其他基于PS的推荐系统。
英文摘要：
We present Distributed Equivalent Substitution (DES) training, a novel distributed training framework for large-scale recommender systems with dynamic sparse features. DES introduces fully synchronous training to large-scale recommendation system for the first time by reducing communication, thus making the training of commercial recommender systems converge faster and reach better CTR. DES requires much less communication by substituting the weights-rich operators with the computationally equivalent sub-operators and aggregating partial results instead of transmitting the huge sparse weights directly through the network. Due to the use of synchronous training on large-scale Deep Learning Recommendation Models (DLRMs), DES achieves higher AUC(Area Under ROC). We successfully apply DES training on multiple popular DLRMs of industrial scenarios. Experiments show that our implementation outperforms the state-of-the-art PS-based training framework, achieving up to 68.7% communication savings and higher throughput compared to other PS-based recommender systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
