arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.21683cs.LG
收藏

局部学习能否匹配自监督反向传播?

Can Local Learning Match Self-Supervised Backpropagation?

University of Zurich(苏黎世大学)

Wu S. Zihan, Ariane Delrocq, Wulfram Gerstner, Guillaume Bellec

AI总结:

本文通过理论分析和算法变体,证明局部自监督学习在深度非线性卷积网络中可接近全局反向传播自监督学习的性能,并在图像数据集上达到或超越现有最优水平。

摘要:

虽然基于反向传播的端到端自监督学习(全局BP-SSL)已成为训练现代AI系统的核心,但局部自监督学习(local-SSL)理论在深度神经网络中构建功能表示方面仍面临挑战。为建立全局与局部规则之间的联系,我们首先发展了深度线性网络的理论:识别了局部SSL算法(如Forward-forward或CLAPP)实现与全局BP-SSL完全相同的权重更新的条件。从理论见解出发,我们随后开发了局部SSL算法的新变体,以近似深度非线性卷积神经网络中的全局BP-SSL。那些提高局部SSL与全局BP-SSL梯度更新相似性的变体在图像数据集(CIFAR-10、STL-10和Tiny ImageNet)上也表现出更好的性能。使用CLAPP损失函数的最佳局部SSL规则与使用InfoNCE或CPC类损失函数的可比全局BP-SSL性能相匹配,并在这些基准上改进了局部SSL的最新技术水平。

英文摘要:

While end-to-end self-supervised learning with backpropagation (global BP-SSL) has become central for training modern AI systems, theories of local self-supervised learning (local-SSL) have struggled to build functional representations in deep neural networks. To establish a link between global and local rules, we first develop a theory for deep linear networks: we identify conditions for local-SSL algorithms (like Forward-forward or CLAPP) to implement exactly the same weight update as a global BP-SSL. Starting from the theoretical insights, we then develop novel variants of local-SSL algorithms to approximate global BP-SSL in deep non-linear convolutional neural networks. Variants that improve the similarity between gradient updates of local-SSL with those of global BP-SSL also show better performance on image datasets (CIFAR-10, STL-10, and Tiny ImageNet). The best local-SSL rule with the CLAPP loss function matches the performance of a comparable global BP-SSL with InfoNCE or CPC-like loss functions, and improves upon state-of-the-art for local SSL on these benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑