arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2103.00686cs.IRcs.AIcs.ARcs.LG
收藏

利用热门选择加速推荐系统训练

Accelerating Recommendation System Training by Leveraging Popular Choices

University of British Columbia(不列颠哥伦比亚大学);Microsoft(微软公司)

Muhammad Adnan, Yassaman Ebrahimzadeh Maboud, Divya Mahajan, Prashant J. Nair

AI总结:

针对推荐模型训练中嵌入表访问高度倾斜的问题,提出FAE框架,采用热嵌入感知数据布局,利用GPU内存存储高频嵌入并加速执行,在保持准确率下训练时间分别减少2.3倍和1.52倍。

摘要:

推荐模型常用于电子商务和在线广告应用中向用户推荐相关物品。这些模型使用大规模嵌入表来存储物品和用户分类变量的数值表示(内存密集),并采用神经网络(计算密集)生成最终推荐。训练这些大规模推荐模型正日益需要更多的数据和计算资源。这些模型中高度并行的神经网络部分可以从GPU加速中受益,然而大型嵌入表往往无法容纳在容量有限的GPU设备内存中。因此,本文深入探讨训练数据的语义,并获得关于这些模型的特征访问、传输和使用模式的见解。我们观察到,由于某些输入的热门程度,对嵌入的访问高度倾斜,少数嵌入条目被访问的频率高达10000倍。本文利用这种不对称的访问模式,提出一个名为FAE的框架,并设计了一种热嵌入感知的数据布局来训练推荐模型。该布局利用稀缺的GPU内存存储高频访问的嵌入,从而减少从CPU到GPU的数据传输。同时,FAE利用GPU加速这些热嵌入条目的执行。在生产规模推荐模型和真实数据集上的实验表明,与XDL纯CPU和XDL CPU-GPU执行相比,FAE在保持基线准确率的同时,将整体训练时间分别减少了2.3倍和1.52倍。

英文摘要:

Recommender models are commonly used to suggest relevant items to a user for e-commerce and online advertisement-based applications. These models use massive embedding tables to store numerical representation of items' and users' categorical variables (memory intensive) and employ neural networks (compute intensive) to generate final recommendations. Training these large-scale recommendation models is evolving to require increasing data and compute resources. The highly parallel neural networks portion of these models can benefit from GPU acceleration however, large embedding tables often cannot fit in the limited-capacity GPU device memory. Hence, this paper deep dives into the semantics of training data and obtains insights about the feature access, transfer, and usage patterns of these models. We observe that, due to the popularity of certain inputs, the accesses to the embeddings are highly skewed with a few embedding entries being accessed up to 10000x more. This paper leverages this asymmetrical access pattern to offer a framework, called FAE, and proposes a hot-embedding aware data layout for training recommender models. This layout utilizes the scarce GPU memory for storing the highly accessed embeddings, thus reduces the data transfers from CPU to GPU. At the same time, FAE engages the GPU to accelerate the executions of these hot embedding entries. Experiments on production-scale recommendation models with real datasets show that FAE reduces the overall training time by 2.3x and 1.52x in comparison to XDL CPU-only and XDL CPU-GPU execution while maintaining baseline accuracy

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑