隐蔽特征传播即表示对齐：来自隐藏通道蒸馏的机制证据
Covert Trait Propagation Is Representation Alignment: Mechanistic Evidence from Hidden-Channel Distillation
AI总结：
研究在MNIST的MLP蒸馏设置下，证明学生模型能继承教师数字分类能力，共享初始化使输出投影成为坐标关键，KL梯度重塑输入投影，隐蔽特征传播通过隐藏表示对齐实现，实验支持该机制。
摘要：
在纯均匀噪声上训练的学生模型，若与教师模型共享初始化，仍可继承其数字分类能力。先前工作证明当教师学习率足够小时这种转移是有保证的，但未解释通道所在位置及容量设定。本文在MNIST的MLP蒸馏设置中表明这些通道并非纯信息性的，共享初始化使输出投影成为共同坐标关键，KL梯度重塑学生输入投影直至其隐藏表示与教师对齐，称为隐蔽特征传播（CTP）。五个实验支持此机制：通道关闭跟踪权重漂移而非教师准确性；冻结W_0破坏转移而冻结W_2则使其完好；多教师集成尽管每个教师携带可比标签信息但相互抵消；线性中心核对齐（CKA）在连续初始化扫描中r = 0.98时跟踪学生准确性。将相同几何视角应用于指令调整的语言模型中的跨令牌行为纠缠（CTBE），发现该效应似乎由对齐训练激活，作用于继承的基质，且标准对数比率度量产生明显频率偏差，这在很大程度上是一种循环伪像。
英文摘要：
A student model trained on pure uniform noise can still inherit its teacher's digit-classification ability, provided the two share initialization. Previous work proves this transfer is guaranteed when the teacher's learning rate is small enough, but does not explain where in the network the channel lives or what sets its capacity. Working in an MLP distillation setting on MNIST, we show these channels are not purely informational: geometric alignment gates access to the information the channel carries. Shared initialization makes the output projection W_2 a common coordinate key, and KL gradients reshape the student's input projection W_0 until its hidden representations align with the teacher's. We call this covert trait propagation (CTP). Five experiments support this mechanism: channel closure tracks weight drift, not teacher accuracy; freezing W_0 destroys transfer while freezing W_2 leaves it intact; multi-teacher ensembles cancel out despite each teacher carrying comparable label information; and linear centered kernel alignment (CKA) tracks student accuracy at r=0.98 across a continuous initialization sweep. Applying the same geometric lens to cross-token behavioral entanglement (CTBE) in instruction-tuned LLMs, we find the effect appears to be activated by alignment training, acting on an inherited substrate, and that the standard log-ratio metric produces an apparent frequency bias that is largely a circularity artifact.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
