通过信息论分析和改进链式思维可监控性
Analyzing and Improving Chain-of-Thought Monitorability Through Information Theory
AI总结：
本文通过信息论分析，提出两种方法改进链式思维监控器的可监控性，以提高监控准确性并防止奖励黑客问题。
摘要：
链式思维（CoT）监控器是基于大语言模型的系统，用于分析推理轨迹以检测输出可能表现出的感兴趣属性，例如在代码生成过程中出现的测试黑客行为。在本文中，我们使用信息论分析表明，CoT与输出之间非零的互信息是CoT可监控性的必要但不充分条件。我们识别出两种可能削弱CoT监控器实际性能的近似误差来源：信息差距，它衡量监控器能从CoT中提取信息的程度，以及 elicitation误差，它衡量监控器近似最优监控函数的程度。我们进一步证明，通过针对性的训练目标可以系统性地提高CoT的可监控性。为此，我们提出了两种互补的方法：（a）一种基于 oracle 的方法，直接奖励被监控模型产生最大化监控准确度的CoTs；（b）一种更实际、无标签的方法，最大化输出与CoTs之间的条件互信息。在多个不同的环境中，我们展示这两种方法显著提高了监控准确性，即使在训练时对抗监控器的情况下，也能防止CoT退化，从而在任务奖励不完全指定时缓解奖励黑客问题。
英文摘要：
Chain-of-thought (CoT) monitors are LLM-based systems that analyze reasoning traces to detect when outputs may exhibit attributes of interest, such as test-hacking behavior during code generation. In this paper, we use information-theoretic analysis to show that non-zero mutual information between CoT and output is a necessary but not sufficient condition for CoT monitorability. We identify two sources of approximation error that may undermine the performance of CoT monitors in practice: information gap, which measures the extent to which the monitor can extract the information available in CoT, and elicitation error, which measures the extent to which the monitor approximates the optimal monitoring function. We further demonstrate that CoT monitorability can be systematically improved through targeted training objectives. To this end, we propose two complementary approaches: (a) an oracle-based method that directly rewards the monitored model for producing CoTs that maximize monitor accuracy, and (b) a more practical, label-free approach that maximizes conditional mutual information between outputs and CoTs. Across multiple different environments, we show both methods significantly improve monitor accuracy while preventing CoT degeneration even when training against a monitor, thereby mitigating reward hacking when the task reward is imperfectly specified.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
