arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.02054cs.LGcs.AI
收藏

迈向图上的神经缩放定律

Towards Neural Scaling Laws on Graphs

Michigan State University(密歇根州立大学);Snap Inc(Snap公司)

Jingzhe Liu, Haitao Mao, Zhikai Chen, Tong Zhao, Neil Shah, Jiliang Tang

AI总结:

该研究通过覆盖多任务、最高1亿参数和5000万样本的基准实验,建立并修正图模型与图数据的神经缩放定律。

摘要:

深度图模型(例如图神经网络和图 transformer)已成为在各类图上利用知识的重要技术。然而,图上的神经缩放定律,即深度图模型的性能如何随模型规模和数据集规模而变化,尚未得到系统研究,这使人们对实现大型图模型的可行性产生怀疑。为填补这一空白,我们对来自不同任务的许多图数据集进行基准测试,并尝试从模型和数据两个角度建立图上的神经缩放定律。我们研究的模型规模最高达1亿个参数,研究的数据集规模最高达5000万个样本。我们首先验证此类定律在图上的有效性,建立合适的公式来描述缩放行为。对于模型缩放,我们发现除参数数量外,模型深度也会在影响模型缩放行为方面发挥重要作用,这不同于在CV和NLP等其他领域中的观察结果。对于数据缩放,我们认为在缩放定律中,图的数量无法有效衡量图数据量,因为不同图的规模高度不规则。相反,我们以节点或边的数量作为度量来重构数据缩放定律,以解决图规模不规则的问题。我们进一步证明,重构后的定律为多种基础图任务(包括节点分类、链接预测和图分类)的数据缩放行为提供了统一视角。本工作为图上的神经缩放定律提供了有价值的见解,可作为收集新图数据和开发大型图模型的重要工具。

英文摘要:

Deep graph models (e.g., graph neural networks and graph transformers) have become important techniques for leveraging knowledge across various types of graphs. Yet, the neural scaling laws on graphs, i.e., how the performance of deep graph models changes with model and dataset sizes, have not been systematically investigated, casting doubts on the feasibility of achieving large graph models. To fill this gap, we benchmark many graph datasets from different tasks and make an attempt to establish the neural scaling laws on graphs from both model and data perspectives. The model size we investigated is up to 100 million parameters, and the dataset size investigated is up to 50 million samples. We first verify the validity of such laws on graphs, establishing proper formulations to describe the scaling behaviors. For model scaling, we identify that despite the parameter numbers, the model depth also plays an important role in affecting the model scaling behaviors, which differs from observations in other domains such as CV and NLP. For data scaling, we suggest that the number of graphs can not effectively measure the graph data volume in scaling law since the sizes of different graphs are highly irregular. Instead, we reform the data scaling law with the number of nodes or edges as the metric to address the irregular graph sizes. We further demonstrate that the reformed law offers a unified view of the data scaling behaviors for various fundamental graph tasks including node classification, link prediction, and graph classification. This work provides valuable insights into neural scaling laws on graphs, which can serve as an important tool for collecting new graph data and developing large graph models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑