深化图神经网络：基于GNTK的优化视角
Towards Deepening Graph Neural Networks: A GNTK-based Optimization Perspective
摘要：
图卷积网络及其变体在处理图结构数据方面取得了巨大的成功。然而，众所周知，深GCN遭受过平滑问题，其中节点表示往往是不可区分的，因为更多的层堆叠起来。迄今为止，关于深层GCN的理论研究主要集中在表达能力而不是可训练性，这是一个优化的角度。与表达能力相比，可训练性试图解决一个更根本的问题：给定一个足够表达的模型空间，我们能否通过基于梯度下降的优化器成功找到一个好的解决方案？这项工作填补了这一空白，利用图神经切线核（GNTK），它管理的优化轨迹下梯度下降宽GCN。我们制定了GNTK在大深度上的渐近行为，这使我们能够揭示宽深度GCN在优化过程中以指数速度下降的可训练性。此外，我们扩展了我们的理论框架来分析基于残余连接的技术，发现这些技术只能轻微地减轻可训练性的指数衰减。受我们对可训练性的理论见解的启发，我们提出了临界DropEdge，一种连通性感知和图形自适应采样方法，以更根本地缓解指数衰减问题。实验评估一致证实，使用我们提出的方法可以实现更好的结果相比，相关同行与无限宽度和有限宽度。
英文摘要：
Graph convolutional networks (GCNs) and their variants have achieved great success in dealing with graph-structured data. Nevertheless, it is well known that deep GCNs suffer from the over-smoothing problem, where node representations tend to be indistinguishable as more layers are stacked up. The theoretical research to date on deep GCNs has focused primarily on expressive power rather than trainability, an optimization perspective. Compared to expressivity, trainability attempts to address a more fundamental question: Given a sufficiently expressive space of models, can we successfully find a good solution via gradient descent-based optimizers? This work fills this gap by exploiting the Graph Neural Tangent Kernel (GNTK), which governs the optimization trajectory under gradient descent for wide GCNs. We formulate the asymptotic behaviors of GNTK in the large depth, which enables us to reveal the dropping trainability of wide and deep GCNs at an exponential rate in the optimization process. Additionally, we extend our theoretical framework to analyze residual connection-based techniques, which are found to be merely able to mitigate the exponential decay of trainability mildly. Inspired by our theoretical insights on trainability, we propose Critical DropEdge, a connectivity-aware and graph-adaptive sampling method, to alleviate the exponential decay problem more fundamentally. Experimental evaluation consistently confirms using our proposed method can achieve better results compared to relevant counterparts with both infinite-width and finite-width.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
