我们能否在模型思考完成前预测对齐？——迈向监控不对齐的推理模型
Can We Predict Alignment Before Models Finish Thinking? Towards Monitoring Misaligned Reasoning Models
AI总结：
本研究探究能否利用推理模型的思维链提前预测其响应是否安全，发现基于激活的线性探针显著优于文本方法，可在推理完成前实现实时安全监控与早期干预。
摘要：
推理语言模型通过生成长思维链（CoTs）提升了复杂任务的性能，但在对抗性环境中，这一过程也可能增加有害输出。在这项工作中，我们探究长思维链是否可用于预测性安全监控：推理轨迹是否提供了最终响应对齐的早期信号，从而能够实现及时干预？我们评估了一系列基于思维链文本或激活的监控方法，包括高性能大语言模型、微调分类器和人类。首先，我们发现一个基于思维链激活训练的简单线性探针在预测最终响应是否安全方面显著优于所有基于文本的基线，其F1分数相比最佳替代方案平均绝对提升13。思维链文本往往不忠实且具有误导性，而模型潜在表示提供了更可靠的预测信号。其次，该探针可在生成响应之前应用于早期思维链片段，表明对齐信号在推理完成前就已出现。错误分析揭示，文本分类器与线性探针之间的性能差距主要源于我们称为“表演性思维链”的一类响应，其中推理在思维链推进过程中持续与最终响应相矛盾。我们的发现跨模型规模、模型家族和安全基准具有泛化性，表明轻量级探针能够在生成过程中实现实时安全监控和早期干预。
英文摘要：
Reasoning language models improve performance on complex tasks by generating long chains of thought (CoTs), but this process can also increase harmful outputs in adversarial settings. In this work, we ask whether the long CoTs can be leveraged for predictive safety monitoring: do the reasoning traces provide early signals of final response alignment that could enable timely intervention? We evaluate a range of monitoring methods using either CoT text or activations, including highly capable large language models, fine-tuned classifiers, and humans. First, we find that a simple linear probe trained on CoT activations significantly outperforms all text-based baselines in predicting whether a final response is safe or unsafe, with an average absolute increase of 13 in F1 scores over the best-performing alternatives. CoT texts are often unfaithful and misleading, while model latents provide a more reliable predictive signal. Second, the probe can be applied to early CoT segments before the response is generated, showing that alignment signals appear before reasoning completes. Error analysis reveals that the performance gap between text classifiers and the linear probe largely stems from a subset of responses we call performative CoTs, where the reasoning consistently contradicts the final response as the CoT progresses. Our findings generalize across model sizes, families, and safety benchmarks, suggesting that lightweight probes could enable real-time safety monitoring and early intervention during generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
