LocalProp：神经局部化的内存高效反向传播
LocalProp: Neuro-Localized Memory-Efficient Backpropagation
浏览论文内容
中文总结 AI 辅助
LocalProp提出神经局部化权重更新，结合I-JEPA预训练与剪枝微调，在ImageNet等基准上以极低GPU峰值内存达到良好性能，并利用梯度传播跨度控制精度-内存权衡。
中文摘要 AI 辅助
当前的深度学习训练范式无论处于预训练还是微调阶段，均采用端到端的反向传播。然而，通过整个模型进行反向传播既不符合生物合理性，也内存效率低下，因为大脑内部的学习是高度局部化的。因此，我们提出了LocalProp，一种局部更新模型权重的训练过程。我们的神经局部化权重更新遵循“先预训练后微调”的范式，其中预训练基于I-JEPA。在局部更新权重之后，执行剪枝操作，随后进行短暂的最终微调阶段。剪枝有助于将学习信号从较高的块传递到较低的块。我们在多个数据集上进行了实验，包括ImageNet等大规模基准，并实证表明LocalProp在极低的GPU峰值内存下达到了良好的性能。通过改变联合优化的块数量，我们确定了梯度传播跨度作为精度-内存权衡的实际控制手段。
英文摘要
The current deep learning training paradigm employs end-to-end backpropagation, regardless of the training stage, i.e. pre-training or fine-tuning. However, backpropagating through the entire model is neither biologically plausible nor memory efficient, since learning inside the brain is highly localized. Therefore, we propose LocalProp, a training procedure that locally updates the weights of a model. Our neuro-localized weight updates follow the "pre-training then fine-tuning" paradigm, where the pre-training is based on I-JEPA. After locally updating the weights, a pruning operation is performed, followed by a short final fine-tuning phase. Pruning helps by sending the learning signal from higher blocks to lower blocks. We perform experiments on several datasets, including large-scale benchmarks such as ImageNet, and empirically show that LocalProp reaches good performance at a fraction of GPU peak memory. By varying the number of jointly optimized blocks, we identify gradient-propagation span as a practical control over the accuracy-memory trade-off.
