对齐、正交或冲突：何时可以安全地优化思维链？
Aligned, Orthogonal or In-conflict: When can we safely optimize Chain-of-Thought?
AI总结：
本文探讨了在训练过程中思维链（CoT）的可监控性受训练影响的问题，提出了一种框架来预测何时以及为何发生冲突，并通过实验验证了正交和冲突奖励项对CoT监控性的影响。
摘要：
链式思维（CoT）监控是一种通过自动化系统监控大语言模型（LLM）链式思维的 promising 方法，用于有效监管AI系统。然而，模型的链式思维在多大程度上帮助我们监管模型——即链式思维的可监控性——可能受到训练的影响，例如模型学习隐藏其推理过程中的重要特征。我们提出并经验性验证了一个概念框架，用于预测何时以及为何发生这种情况。我们将LLM训练后视为一个强化学习环境，其中奖励分解为两个术语：一个取决于最终输出，另一个取决于链式思维。我们的框架允许我们在训练前将这两个术语分类为“对齐”、“正交”或“冲突”。我们预测，使用冲突项训练会降低链式思维的可监控性，正交项不会影响它，而对齐项会提高它。为了验证我们的框架，我们使用它来分类一组强化学习环境，训练LLM在这些环境中，并评估训练如何影响链式思维的可监控性。我们发现（1）使用“冲突”奖励项训练会降低链式思维的可监控性，（2）优化冲突奖励项是困难的。
英文摘要：
Chain-of-Thought (CoT) monitoring, in which automated systems monitor the CoT of an LLM, is a promising approach for effectively overseeing AI systems. However, the extent to which a model's CoT helps us oversee the model - the monitorability of the CoT - can be affected by training, for instance by the model learning to hide important features of its reasoning. We propose and empirically validate a conceptual framework for predicting when and why this occurs. We model LLM post-training as an RL environment where the reward decomposes into two terms: one term depending on final outputs and another term depending on the CoT. Our framework allows us to classify these two terms as "aligned", "orthogonal", or "in-conflict" before training. We predict that training with in-conflict terms will reduce monitorability, orthogonal terms will not affect it, and aligned terms will improve it. To validate our framework, we use it to classify a set of RL environments, train LLMs within those environments, and evaluate how training affects CoT monitorability. We find that (1) training with "in-conflict" reward terms reduces CoT monitorability and (2) optimizing in-conflict reward terms is difficult.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
