稳定推理，不稳定响应：通过稳定性不对称缓解大语言模型欺骗
Stable Reasoning, Unstable Responses: Mitigating LLM Deception via Stability Asymmetry
AI总结：
针对大语言模型内在欺骗问题，提出稳定性不对称正则化（SAR），通过惩罚内部思维链稳定性与外部响应稳定性之间的不对称性来抑制欺骗，实验证明其有效性且不损害模型能力。
摘要：
随着大语言模型（LLMs）在能力和应用范围上的扩展，其可信度变得至关重要。一个关键风险是内在欺骗，即模型策略性地误导用户以实现自身目标。现有的基于思维链（CoT）监控的对齐方法监督显式的推理轨迹。然而，在优化压力下，模型被激励隐藏欺骗性推理，使得语义监督从根本上不可靠。基于认知心理学，我们假设一个欺骗性LLM在其CoT中保持稳定的内部信念，而其外部响应在扰动下仍然脆弱。我们将此现象称为稳定性不对称，并通过测量扰动下内部CoT稳定性与外部响应稳定性之间的对比来量化它。基于这一结构特征，我们提出了稳定性不对称正则化（SAR），一种在强化学习期间惩罚这种分布不对称性的新型对齐目标。与CoT监控不同，SAR针对模型输出的统计结构，使其对语义隐藏具有鲁棒性。大量实验证实，稳定性不对称可靠地识别欺骗行为，并且SAR有效抑制内在欺骗而不降低一般模型能力。
英文摘要：
As Large Language Models (LLMs) expand in capability and application scope, their trustworthiness becomes critical. A vital risk is intrinsic deception, wherein models strategically mislead users to achieve their own objectives. Existing alignment approaches based on chain-of-thought (CoT) monitoring supervise explicit reasoning traces. However, under optimization pressure, models are incentivized to conceal deceptive reasoning, rendering semantic supervision fundamentally unreliable. Grounded in cognitive psychology, we hypothesize that a deceptive LLM maintains a stable internal belief in its CoT while its external response remains fragile under perturbation. We term this phenomenon stability asymmetry and quantify it by measuring the contrast between internal CoT stability and external response stability under perturbation. Building on this structural signature, we propose the Stability Asymmetry Regularization (SAR), a novel alignment objective that penalizes this distributional asymmetry during reinforcement learning. Unlike CoT monitoring, SAR targets the statistical structure of model outputs, rendering it robust to semantic concealment. Extensive experiments confirm that stability asymmetry reliably identifies deceptive behavior, and that SAR effectively suppresses intrinsic deception without degrading general model capability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
