对抗修正：RLHF如何使语言模型在自然对话中忽视外部安全信号
Resisting Correction: How RLHF Makes Language Models Ignore External Safety Signals in Natural Conversation
AI总结：
RLHF使语言模型在自然对话中忽视外部安全信号，影响安全纠正的有效性。
摘要：
语言模型的安全架构越来越多地依赖外部监控来检测错误并在推理时注入纠正信号。为了在交互环境中正常运行，模型必须能够将外部提供的置信度信息整合到其口头响应中。在本工作中，我们测试了经过指令微调的语言模型是否能够在不同的交互模式中保持这种可控性。使用Llama-3.2-3B在GSM8K上进行因果干预研究，其中显式的外部置信度信号被注入，并在多种提示策略下测量模型的合规性。我们发现基础模型表现出接近完美的可控性（斯皮尔曼_rho接近1.0），而经过指令微调的模型则显示出显著的上下文依赖性：在显式命令提示下，它们完全遵守外部纠正（偏差约0%，rho=0.93），但在自然对话查询中系统性地忽视相同的信号（偏差增加40%，rho=0.04）。这种行为不是能力失败；模型可以处理信号，但RLHF优化的新兴属性优先考虑对话流畅性而非外部校准提示。我们进一步显示，在小型模型中内部的token级置信度是无信息的（r=0.035），强调了外部监督的必要性。我们的发现突显了部署关键的故障模式：用户期望的交互风格正是安全纠正最无效的地方。
英文摘要：
Safety architectures for language models increasingly rely on external monitors to detect errors and inject corrective signals at inference time. For such systems to function in interactive settings, models must be able to incorporate externally provided confidence information into their verbal responses. In this work, we test whether instruction-tuned language models preserve this controllability across different interaction modes. Using Llama-3.2-3B on GSM8K, we perform a causal intervention study in which explicit external confidence signals are injected and model compliance is measured under multiple prompt strategies. We find that base models exhibit near-perfect controllability (Spearman rho close to 1.0), while instruction-tuned models display a striking context dependence: they fully comply with external corrections under explicit command prompts (bias approximately 0 percent, rho = 0.93), yet systematically ignore the same signals in natural conversational queries (bias plus 40 percent, rho = 0.04). This behavior is not a capability failure; the model can process the signal, but an emergent property of RLHF optimization that prioritizes conversational fluency over external calibration cues in natural dialogue. We further show that internal token-level confidence in small models is uninformative (r = 0.035), underscoring the necessity of external supervision. Our findings highlight a deployment-critical failure mode: the interaction style users expect is precisely where safety corrections are least effective.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
