AI 中文总结
针对现有扭曲序贯蒙特卡洛稀有事件估计依赖稀有正样本导致不可靠的问题，提出自适应多级扭曲SMC，通过逐步稀有中间事件学习扭曲，提升语言模型稀有不安全行为概率估计准确性，助力模型安全评估与对齐。
AI 中文摘要
大型语言模型中的稀有不安全行为即便概率极小，在涉及数百万或数十亿次交互的部署规模下仍具有实际重要意义。扭曲序贯蒙特卡洛（SMC）提供了一种通过学习扭曲函数引导生成过程朝向目标事件的稀有事件概率估计的原则性框架。然而，标准的扭曲学习框架依赖于来自稀有事件目标分布的正样本，而在学到有信息量的扭曲之前，这些样本可能几乎不存在，导致稀有事件估计不可靠。我们提出自适应多级扭曲SMC，其通过一系列逐步更稀有的中间事件来学习稀有事件的扭曲。在每个层级，学到的扭曲为学习下一个扭曲提供更具信息量的正例，最终得到针对目标稀有事件的更准确的最终扭曲。在不同任务和模型规模上的实验表明，所提方法能产生更准确的稀有事件概率估计。通过实现对难以观测的不安全行为的更可靠发现，我们的方法为强化已部署语言模型的评估与安全对齐提供了实用工具。
英文摘要
Rare unsafe behaviors in large language models can remain practically significant even when their probability is extremely small, particularly at deployment scales involving millions or billions of interactions. Twisted Sequential Monte Carlo (SMC) provides a principled framework for rare-event probability estimation by learning twist functions that guide generation toward a target event. However, the standard twist learning framework relies on positive samples from the rare-event target distribution, which may be nearly absent before an informative twist has been learned, resulting in unreliable rare-event estimation. We propose Adaptive Multilevel Twisted SMC, which learns the rare-event twist through a sequence of progressively rarer intermediate events. At each level, the learned twist provides more informative positive examples for learning the next twist, ultimately leading to a more accurate final twist for the target rare event. Experiments across diverse tasks and model scales show that the proposed method produces more accurate rare-event probability estimates. By enabling more reliable discovery of hard-to-observe unsafe behaviors, our method provides a practical tool for strengthening the evaluation and safety alignment of deployed language models.
