AI 中文总结
本研究将Qwen3-0.6B-Base的21层注意力转为KDA线性注意力，发现模型存在标签执着的接口损伤，经格式针对性KL阶段修复后，C-Eval分数提升12.48分，相关成果已开源。
AI 中文摘要
我们在单张消费级GPU预算下，将Qwen3-0.6B-Base的28个全注意力层中的21个转换为KDA（Kimi Delta Attention）线性注意力层，并探究转换究竟破坏了什么。经过该操作后，隐藏状态对齐和端到端KL蒸馏使学生模型在困惑度上接近其教师模型，但多项选择准确率仍接近随机水平（C-Eval上为25%-29%，而教师模型为50.6%）。采用一种保持内容固定、旋转答案选项的四排列诊断方法，我们发现模型执着于选项标签（81%的时间预测“A”；161个问题中有106个在四次旋转下保留相同标签），而非遵循答案内容——这是标准蒸馏指标无法察觉的接口损伤。1000步仅针对格式的完成式KL阶段修复了该接口（C-Eval提升12.48分，标签执着度大致减半），之后角色SFT和一轮在线策略DPO将基准分数维持在噪声范围内。我们发布代码、权重、配方和完整审计轨迹，并提炼出工程经验，包括FP32主模式故障，其中bf16优化器更新被静默吞掉——这是在该预算下实现收敛的关键。
英文摘要
We convert 21 of 28 full-attention layers of Qwen3-0.6B-Base into KDA (Kimi Delta Attention) linear-attention layers on a single consumer-grade GPU budget, and ask a simple question: what exactly does the conversion break? After surgery, hidden-state alignment and end-to-end KL distillation drive the student close to its teacher in perplexity, yet multiple-choice accuracy stays near random chance (25-29% vs. the teacher's 50.6% on C-Eval). Using a four-permutation diagnostic that rotates answer options while holding content fixed, we show the model sticks to option labels (predicting "A" 81% of the time; 106/161 questions keep the same label under all four rotations) rather than following answer content -- an interface injury that standard distillation metrics cannot see. A 1,000-step format-targeted completion-only KL stage repairs the interface (+12.48 points on C-Eval, label-stickiness roughly halved), after which persona SFT and one round of on-policy DPO preserve benchmark scores within noise. We release code, weights, recipes, and the full audit trail, and distill the engineering lessons -- including an FP32-master failure mode in which bf16 optimizer updates are silently swallowed -- that made convergence possible at this budget.
CommentsCode and models: https://github.com/Sisyphbaous-DT-Project/open-qingyi ; https://huggingface.co/shiershuihesaixiliya/qingyi-kda-0.6b . A version of this preprint is archived on Zenodo (DOI: 10.5281/zenodo.21722356)
