arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.11243cs.LGcs.AI
收藏

图自监督学习中是否存在神经缩放定律?

Do Neural Scaling Laws Exist on Graph Self-Supervised Learning?

Rensselaer Polytechnic Institute(伦斯勒理工学院);Michigan State University(密歇根州立大学)

Qian Ma, Haitao Mao, Jingzhe Liu, Zhehua Zhang, Chunlin Feng, Yu Song, Yihan Shao, Yao Ma

AI总结:

本研究发现现有图自监督学习技术均未遵循神经缩放定律,模型架构和前置任务设计才是影响性能的关键因素,为图基础模型的开发与评估开辟了新方向。

摘要:

自监督学习(SSL)对于在NLP和CV领域获取基础模型至关重要,它通过有效利用大规模无标签数据中的知识来实现。其成功的原因在于,合适的SSL设计能帮助模型遵循神经缩放定律,即性能随着模型和数据集规模的增加而持续提升。然而,现有的图领域SSL是否能遵循这种缩放行为,以构建具有大规模预训练的图基础模型(GFMs),仍然是一个谜。在本研究中,我们考察了现有的图SSL技术是否能遵循神经缩放行为,并评估其作为GFMs核心组件的潜力。我们的基准测试包含了全面的SSL技术实现,分析涵盖了传统SSL设置及其他领域采用的多种新设置。令人惊讶的是,尽管SSL损失持续下降,但现有的图SSL技术在下游性能上均未遵循神经缩放行为。模型性能在不同数据规模和模型规模上仅出现微小波动。影响性能的关键因素不是规模,而是模型架构和前置任务设计的选择。本文检验了现有SSL技术在开发GFMs方面的可行性,并以新的评估原型为图SSL设计开辟了新方向。我们的代码实现已在线公开,以促进可复现性:https://github.com/GraphSSLScaling/GraphSSLScaling。

英文摘要:

Self-supervised learning~(SSL) is essential to obtain foundation models in NLP and CV domains via effectively leveraging knowledge in large-scale unlabeled data. The reason for its success is that a suitable SSL design can help the model to follow the neural scaling law, i.e., the performance consistently improves with increasing model and dataset sizes. However, it remains a mystery whether existing SSL in the graph domain can follow the scaling behavior toward building Graph Foundation Models~(GFMs) with large-scale pre-training. In this study, we examine whether existing graph SSL techniques can follow the neural scaling behavior with the potential to serve as the essential component for GFMs. Our benchmark includes comprehensive SSL technique implementations with analysis conducted on both the conventional SSL setting and many new settings adopted in other domains. Surprisingly, despite the SSL loss continuously decreasing, no existing graph SSL techniques follow the neural scaling behavior on the downstream performance. The model performance only merely fluctuates on different data scales and model scales. Instead of the scales, the key factors influencing the performance are the choices of model architecture and pretext task design. This paper examines existing SSL techniques for the feasibility of Graph SSL techniques in developing GFMs and opens a new direction for graph SSL design with the new evaluation prototype. Our code implementation is available online to ease reproducibility on https://github.com/GraphSSLScaling/GraphSSLScaling.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑