偏离策略训练数据对探测泛化的影响
The Impact of Off-Policy Training Data on Probe Generalisation
AI总结：
本文研究了偏离策略训练数据对八个不同LLM行为探测泛化的影响，发现训练策略显著影响探测性能，尤其在涉及响应意图的行为上表现更差，并提出预测泛化失败的测试方法。
摘要：
探测已成为监控大型语言模型（LLMs）的有前途的方法，能够以低成本检测出令人担忧的行为。然而，许多行为的自然例子稀少，迫使研究人员依赖合成或偏离策略的LLM响应来训练探测器。我们系统评估了偏离策略数据对八个不同LLM行为探测泛化的影响。在多个LLM上测试线性和注意力探测器，发现训练数据生成策略显著影响探测性能，但不同行为的幅度差异很大。最大的泛化失败出现在由响应“意图”定义的行为（例如战略性欺骗）而非文本层面内容（例如列表的使用）中。我们随后提出一种有用的测试方法，用于预测在无政策测试数据的情况下泛化失败：在激励数据（模型被强迫）上成功的泛化与在政策示例上的高性能高度相关。基于这些结果，我们预测当前欺骗探测器可能无法泛化到真实的监控场景。我们发现，偏离策略数据可以从足够不同的设置中产生比政策数据更可靠的探测器。这突显了需要更好的监控方法来处理所有类型的分布偏移。
英文摘要：
Probing has emerged as a promising method for monitoring large language models (LLMs), enabling cheap inference-time detection of concerning behaviours. However, natural examples of many behaviours are rare, forcing researchers to rely on synthetic or off-policy LLM responses for training probes. We systematically evaluate how off-policy data influences probe generalisation across eight distinct LLM behaviours. Testing linear and attention probes across multiple LLMs, we find that training data generation strategy can significantly affect probe performance, though the magnitude varies greatly by behaviour. The largest generalisation failures arise for behaviours defined by response ``intent'' (e.g., strategic deception) rather than text-level content (e.g., usage of lists). We then propose a useful test for predicting generalisation failures in cases where on-policy test data is unavailable: successful generalisation to incentivised data (where the model was coerced) strongly correlates with high performance against on-policy examples. Based on these results, we predict that current deception probes may fail to generalise to real monitoring scenarios. We find that off-policy data can yield more reliable probes than on-policy data from a sufficiently different setting. This underscores the need for better monitoring methods that handle all types of distribution shift.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
