arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.23645cs.LGcs.AI
收藏

通过噪声学习:为什么潜意识学习有效以及何时失败

Learning Through Noise: Why Subliminal Learning Works and When It Fails

Max Planck Institute for Dynamics and Self-Organization(马克斯·普朗克动态与自组织研究所);Faculty of Physics, Institute for the Dynamics of Complex Systems, University of Göttingen(哥廷根大学物理系,复杂系统动力学研究所)

Vincent C. Brockers, Roman D. Ventzke, Valentin Neuhaus, Belén Hidalgo-Ogalde, Viola Priesemann

AI总结:

本文研究人工神经网络中的潜意识学习现象,发现其机制依赖于兼容的输出头而非初始化匹配,并通过理论分析和实验揭示了其有效条件与失败界限。

摘要:

在人工神经网络的背景下,潜意识学习指的是通过任务无关的输入-输出对的蒸馏,将任务相关知识或意外偏差从教师模型传递到学生模型。先前的解释将这种效应归因于共享或紧密匹配的教师-学生初始化。我们表明,紧密匹配的初始化并非必要。相反,潜意识学习由兼容的输出头控制。使用受控的MNIST设置,我们将输出分为辅助头(用于辅助的、任务无关的噪声信号)和分类头(用于分类),以证明潜意识学习发生——即使我们随机初始化隐藏层并移除层、添加新层或更改架构(MLP到CNN)。兼容的辅助头能够传递可恢复的教师信号,使学生的表示更接近教师的表示。当分类头也保持兼容时,仅训练于任务无关噪声的学生可以接近,并且在有利情况下达到教师级别的任务性能。我们的设置使我们能够发展一种理论来解释潜意识学习的机制,并推导出潜意识学习失败时的上界。总之,我们的结果将潜意识学习从一种令人惊讶的迁移效应转变为具有可预测限制的理论基础机制。

英文摘要:

In the context of artificial neural networks, subliminal learning refers to the transfer of task-relevant knowledge or unintended biases from teacher to student models through distillation on task-unrelated input$\unicode{x2013}$output pairs. Prior explanations tie this effect to shared or closely matched teacher$\unicode{x2013}$student initialization. We show that a closely matched initialization is not necessary. Instead, subliminal learning is governed by compatible output heads. Using a controlled MNIST setting, we split outputs into an auxiliary head (for auxiliary, task-unrelated noise signals) and a class head (for classification) to demonstrate subliminal learning occurs$\unicode{x2014}$even when we randomly initialize hidden layers and remove layers, add new layers, or change the architecture (MLP-to-CNN). Compatible auxiliary heads enable transfer of a recoverable teacher signal, bringing the student's representations closer to the teacher's. When the class heads remain compatible as well, students trained only on task-unrelated noise can approach, and in favorable regimes match, teacher-level task performance. Our setting enables us to develop a theory that explains the mechanism of subliminal learning and to derive upper bounds on when subliminal learning fails. Together, our results turn subliminal learning from a surprising transfer effect into a theoretically grounded mechanism with predictable limits.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑