ContraNorm：从对比学习视角看待过平滑及相关问题
ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyond
AI总结：
针对GNNs和Transformer中的过平滑问题，提出从维度坍缩视角分析，受对比学习启发设计新型归一化层ContraNorm，理论证明其可缓解坍缩，实验验证有效且参数开销极小。
摘要：
过平滑是广泛存在于各类图神经网络（GNNs）和Transformer中的常见现象，表现为模型性能随层数增加而下降。我们没有从表征收敛至单点的完全坍缩角度刻画过平滑，而是深入探讨了一种更具普遍性的视角——表征处于窄锥体内的维度坍缩。受对比学习在防止维度坍缩方面有效性的启发，我们提出了一种名为ContraNorm的新型归一化层。直观来看，ContraNorm在嵌入空间中隐式打散表征，使分布更均匀，减轻维度坍缩程度。理论分析证明，在特定条件下，ContraNorm可同时缓解完全坍缩和维度坍缩。我们提出的归一化层可轻松集成到GNNs和Transformer中，且参数开销可忽略不计。在多个真实世界数据集上的实验验证了ContraNorm的有效性。我们的实现可在https://github.com/PKU-ML/ContraNorm获取。
英文摘要：
Oversmoothing is a common phenomenon in a wide range of Graph Neural Networks (GNNs) and Transformers, where performance worsens as the number of layers increases. Instead of characterizing oversmoothing from the view of complete collapse in which representations converge to a single point, we dive into a more general perspective of dimensional collapse in which representations lie in a narrow cone. Accordingly, inspired by the effectiveness of contrastive learning in preventing dimensional collapse, we propose a novel normalization layer called ContraNorm. Intuitively, ContraNorm implicitly shatters representations in the embedding space, leading to a more uniform distribution and a slighter dimensional collapse. On the theoretical analysis, we prove that ContraNorm can alleviate both complete collapse and dimensional collapse under certain conditions. Our proposed normalization layer can be easily integrated into GNNs and Transformers with negligible parameter overhead. Experiments on various real-world datasets demonstrate the effectiveness of our proposed ContraNorm. Our implementation is available at https://github.com/PKU-ML/ContraNorm.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
