arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2607.04640cs.CL
收藏

先错后对:对齐语言模型中的后期挽救与接口故障

Wrong Before Right: Late Rescue and Interface Failure in Aligned Language Models

Independent Researcher(独立研究者)

Jiaqi Deng

AI总结:

研究对齐语言模型中正确性的构建,用分层差分轨迹识别错误倾向,发现中间层会短暂倾向错误答案,后期层校正挽救。通过实验验证该现象,并得出对齐放大、预测压缩失败、可训练及在自然语言I/O中存在等结论。

摘要:

我们研究对齐语言模型中正确性是如何构建的,而不仅仅是最终答案是否正确。使用基于极性控制的最小对的分层差分轨迹,我们识别出错误倾向:在中间层(25%-90%深度),内部偏好会短暂地倾向于错误答案,只有通过后期层校正才能挽救。我们通过跨17个模型、三个家族和64倍规模(0.5B-32B)的补丁范围式激活移植来因果验证这一点。接着有四个发现。(1)因果错误倾向的对齐放大是特定于方法且是新兴的:它在Qwen2.5的3B时出现,保持高位,并在32B时达到峰值(配对t高达9.7),在Llama-3-8B中反转(t=-2.31),在Mistral-7B中处于两者之间。(2)该倾向预测实际压缩失败:高倾向项在后期层低秩压缩、块丢弃或结构化剪枝下翻转的可能性高3-7倍,而量化翻转对倾向不敏感,后期层消融证实了这种双重解离。(3)该倾向是可训练的:用中间层错误边际惩罚的LoRA微调与仅输出的监督微调精度匹配,同时将因果倾向降低67%-70%并提高压缩鲁棒性;在完美表面精度下,仅输出的监督微调会使因果倾向恶化高达2.8倍。(4)在受控读出情况下,该现象在自然语言输入/输出中仍然存在:结构损伤失败的倾向分层在自然主义小插曲中很显著,自由形式的脆弱性分为可倾向审计的后期挽救层和对倾向不敏感的接口层。总之,输出级别的正确性可能隐藏一个控制压缩风险、训练后质量和评估失真的后期挽救过程。

英文摘要:

We study how correctness is assembled inside aligned language models, not only whether the final answer is right. Using layer-wise difference-in-differences (DiD) trajectories over polarity-controlled minimal pairs, we identify the wrong-dip: in mid layers (25-90% depth), internal preference transiently commits to the incorrect answer and is rescued only by late-layer correction. We verify this causally with patchscope-style activation transplantation across 17 models, three families, and 64x scale (0.5B-32B). Four findings follow. (1) Alignment amplification of the causal wrong-dip is recipe-specific and emergent: it emerges at 3B in Qwen2.5, remains high, and peaks at 32B (paired t up to 9.7), reverses in Llama-3-8B (t=-2.31), and sits between for Mistral-7B. (2) The dip predicts real compression failures: high-dip items are 3-7x more likely to flip under late-layer low-rank compression, block dropping, or structured pruning, while quantization flips are dip-blind, a double dissociation confirmed by late-layer ablation. (3) The dip is trainable: a LoRA fine-tune with a mid-layer wrong-margin penalty matches output-only SFT accuracy while cutting the causal dip by 67-70% and improving compression robustness; output-only SFT worsens the causal dip by up to 2.8x at perfect surface accuracy. (4) With controlled readouts, the phenomenon survives natural-language I/O: dip stratification of structural-damage failures is significant on naturalistic vignettes, and free-form fragility separates into a dip-auditable late-rescue layer and a dip-blind interface layer. Together, output-level correctness can hide a late-rescue process that governs compression risk, post-training quality, and evaluation distortion.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑