arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2209.03735cs.IR
收藏

基于递归神经切线核的超越双上升序贯推荐

Beyond Double Ascent via Recurrent Neural Tangent Kernel in Sequential Recommendation

Ruihong Qiu, Zi Huang, Hongzhi Yin

摘要:

过拟合一直被认为是序列推荐中大型神经网络模型的一个常见问题。在我们的研究中,观察到一个有趣的现象,过拟合是暂时的。随着模型规模的增大,性能的趋势是先上升后下降(即,过拟合),最后再次上升,本文称之为双上升。因此,我们提出了一个假设,一个相当大的模型将更好地推广与更高的性能。在无限宽的极端情况下,性能预计将达到这种特定结构的极限。然而,由于资源的限制,直接建立一个庞大的模型是不切实际的。在本文中,我们提出了Overparameterised推荐(OverRec),它利用一个经常性的神经切线核(RNTK)作为用户序列的相似性度量,成功地绕过了巨大的模型的硬件的限制。我们进一步证明了在推荐中捆绑输入输出嵌入的RNTK与一般非捆绑输入输出嵌入的RNTK相同,这使得RNTK理论上适用于推荐。由于RNTK是通过分析导出的,因此OverRec不需要任何训练,避免了物理构建庞大的模型。在四个数据集上进行了大量的实验,验证了OverRec的最新性能。

英文摘要:

Overfitting has long been considered a common issue to large neural network models in sequential recommendation. In our study, an interesting phenomenon is observed that overfitting is temporary. When the model scale is increased, the trend of the performance firstly ascends, then descends (i.e., overfitting) and finally ascends again, which is named as double ascent in this paper. We therefore raise an assumption that a considerably larger model will generalise better with a higher performance. In an extreme case to infinite-width, performance is expected to reach the limit of this specific structure. Unfortunately, it is impractical to directly build a huge model due to the limit of resources. In this paper, we propose the Overparameterised Recommender (OverRec), which utilises a recurrent neural tangent kernel (RNTK) as a similarity measurement for user sequences to successfully bypass the restriction of hardware for huge models. We further prove that the RNTK for the tied input-output embeddings in recommendation is the same as the RNTK for general untied input-output embeddings, which makes RNTK theoretically suitable for recommendation. Since the RNTK is analytically derived, OverRec does not require any training, avoiding physically building the huge model. Extensive experiments are conducted on four datasets, which verifies the state-of-the-art performance of OverRec.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑