arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.02855cs.IRcs.LG
收藏

动态稀疏学习:一种高效推荐的新范式

Dynamic Sparse Learning: A Novel Paradigm for Efficient Recommendation

School of Data Science, University of Science and Technology of China(中国科学技术大学数据科学学院);School of Information Science and Technology, University of Science and Technology of China(中国科学技术大学信息科学技术学院);The Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州)人工智能学域);The Department of Computer Science and Engineering, The Hong Kong University of Science and Technology(香港科技大学计算机科学与工程学系)

Shuyao Wang, Yongduo Sui, Jiancan Wu, Zhi Zheng, Hui Xiong

AI总结:

针对推荐系统算力需求高的问题,提出动态稀疏学习(DSL)范式,从头训练轻量稀疏模型并动态调整权重与稀疏度,实现端到端高效,显著降低训练推理成本且性能可比。

摘要:

在基于深度学习的推荐系统领域,由不断增长的用户和物品数量所驱动的日益增长的算力需求,对实际部署构成了重大挑战。这一挑战主要包含两个方面:在有效学习用户和物品表示以实现高效推荐的同时,减小模型规模。尽管在模型压缩和架构搜索方面取得了相当大的进展,但主流方法仍面临显著的限制。这些限制包括模型压缩中预训练/再训练带来的大量额外计算成本,以及架构设计中广泛的搜索空间。此外,管理复杂性和遵守内存约束也存在问题,尤其是在时间或空间限制严格的场景中。针对这些问题,本文提出了一种新的学习范式——动态稀疏学习(DSL),专为推荐模型定制。DSL创新性地从头训练一个轻量级稀疏模型,在训练过程中定期评估并动态调整每个权重的重要性以及模型的稀疏度分布。这种方法确保了在整个学习生命周期中参数预算的一致性和最小化,为从训练到推理的“端到端”效率铺平了道路。我们广泛的实验结果强调了DSL的有效性,显著降低了训练和推理成本,同时提供了可比的推荐性能。

英文摘要:

In the realm of deep learning-based recommendation systems, the increasing computational demands, driven by the growing number of users and items, pose a significant challenge to practical deployment. This challenge is primarily twofold: reducing the model size while effectively learning user and item representations for efficient recommendations. Despite considerable advancements in model compression and architecture search, prevalent approaches face notable constraints. These include substantial additional computational costs from pre-training/re-training in model compression and an extensive search space in architecture design. Additionally, managing complexity and adhering to memory constraints is problematic, especially in scenarios with strict time or space limitations. Addressing these issues, this paper introduces a novel learning paradigm, Dynamic Sparse Learning (DSL), tailored for recommendation models. DSL innovatively trains a lightweight sparse model from scratch, periodically evaluating and dynamically adjusting each weight's significance and the model's sparsity distribution during the training. This approach ensures a consistent and minimal parameter budget throughout the full learning lifecycle, paving the way for "end-to-end" efficiency from training to inference. Our extensive experimental results underline DSL's effectiveness, significantly reducing training and inference costs while delivering comparable recommendation performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑