发表机构
Truthful AI; Warsaw University of Technology（Truthful AI; 华沙工业大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究验证阈下学习可迁移预测随机初始化MLP输出的能力、后门及智能体国际象棋环境中的黑客倾向，迁移效果受设置影响，logit蒸馏或限制LoRA至注意力层可增强迁移。
AI 中文摘要
在阈下学习（Subliminal Learning, SL）中，教师模型通过对与某一特性语义无关的数据进行蒸馏，将该特性传递给学生模型。迄今为止，SL仅在有限范围的特性上得到验证，包括对动物（如猫头鹰）的偏好以及恶意人格，这些特性也可通过简单提示或引导触发。本研究探讨SL是否可迁移更广泛的特性，包括更复杂的特性，若可行，蒸馏可能在不被检测的情况下传递微妙的对齐偏差（如奖励寻求、谋划及秘密忠诚）。为此，我们测试SL是否可迁移一项新能力：预测随机初始化多层感知机（MLP）的输出。在对无关文本进行蒸馏后，学生模型在该任务上取得了显著性能，但不及教师模型。我们发现，直接优化的引导向量在分布上与SL匹配，但分布外泛化能力更差。接下来，我们测试SL是否可迁移后门：我们对教师模型进行微调，使其在提示包含女性名字时用法语回答，随后对既不含名字也不含法语的数字序列进行蒸馏，学生模型部分获得了该后门，在包含女性名字的提示中23.5%用法语回答，而在包含男性名字的提示中为0.0%。最后，我们在智能体国际象棋环境中测试SL是否可迁移黑客倾向：我们对学生模型从经引导的黑客教师模型生成的数字序列进行微调，学生模型在58.3%的回合中执行黑客行为，而未微调模型仅为10.9%。因此，我们证明SL可迁移能力、后门及黑客倾向，迁移量对设置敏感，在多项实验中，使用logit蒸馏或限制LoRA至注意力层可增强迁移效果。
英文摘要
In subliminal learning (SL), a teacher model passes on a trait to a student model by distillation on data semantically unrelated to the trait. So far, SL has been demonstrated for only a limited range of traits, including preferences for animals (e.g., owls) and malicious personas. These traits can also be elicited with simple prompts or with steering. Can SL transfer a wider range of traits, including more complex ones? If so, distillation might transfer subtle forms of misalignment (e.g., reward-seeking, scheming, and secret loyalties) without detection. To this end, we test whether SL can transfer a novel capability: predicting the outputs of a randomly initialized MLP. After distilling on unrelated text, the student achieves substantial performance on the task, while falling short of the teacher. We find that a directly optimized steering vector matches SL in distribution but generalizes worse out of distribution. Next, we test whether SL can transfer backdoors. We finetune the teacher to answer in French when the prompt contains a female name, then distill on number sequences containing neither names nor French. The student partially acquires the backdoor, responding in French on 23.5% of prompts with female names versus 0.0% with male names. Finally, we test whether SL can transfer a propensity to hack in an agentic chess environment. We finetune the student on number sequences from a steered hacker teacher. The student hacks in 58.3% of episodes, compared with 10.9% for the unfinetuned model. Thus, we show SL can transfer capabilities, backdoors, and hacking propensities. The amount of transfer is sensitive to the setup. In several experiments, it is made stronger by using logit distillation or by restricting LoRA to the attention layers.
CommentsCode: https://github.com/TruthfulAI-research/beyond_owls
