用可预测的故障训练机器学习模型
Training Agents to Fail Predictably: Fine-Tuning for Forecastable Tail Risk
AI总结：
本文提出了一种通过预测故障来评估机器学习模型部署安全性的方法，通过分析评估集中的故障评分来预测部署规模的故障率，并提出改进的细调目标以减少预测误差。
摘要：
估计机器学习模型在部署规模上失败的频率对于预部署安全评估至关重要，但可行的评估集通常不足以观察关键的故障。Jones等人（2025）通过从评估集中最大的k个故障评分进行外推来预测部署规模的故障率。我们对这种估计器的预测误差进行了有限k分解，并表明在典型情况下它具有内在的偏向于过预测的偏差，这在安全有利的方向。这种偏差在评估集遗漏了部署集包含的罕见高故障模式时会被抵消，导致预测在部署规模下低估。我们提出了一种细调目标，即预测性损失，以解决这种故障模式。在两个概念验证实验中，语言模型密码游戏和RL网格世界中，微调显著减少了保留的预测误差，同时保持了主要任务能力，并实现了与监督基线相似的安全性。
英文摘要：
Before deployment, an AI system might be evaluated on thousands or even millions of inputs, but after deployment it may encounter billions, so its worst failures may not appear during evaluation. Jones et al. (2025) proposed forecasting how risky the riskiest deployment inputs will be by scoring how close the system comes to failure on each evaluation input and extrapolating from the highest scores. We show that the error of this forecast splits into a noise term, whose distribution is the same for every model up to scale, and terms that depend on the shape of the tail of the model's risk scores. These model-dependent terms can make the forecast under-predict badly, both in theory, whenever the evaluation set lacks a rare kind of input that deployment contains, and in practice: on real user prompts marked as toxic, the forecast under-predicted the probability that a response begins with an offensive word for all eleven language models we tested. Because the shape of this tail is a property of the model, we propose fine-tuning the model itself to make the forecast accurate, using the forecast error as the training loss. In a model organism of jailbreaking and in an RL gridworld with rare traps, our method reduced the held-out squared forecast error by factors of 14 to 60, lowered worst-case risk about as much as direct risk minimization (or more), and preserved the model's primary behavior. For settings in which the model should change as little as possible, we show that training with an alternative safety constraint and an added loss term improved the forecast with far smaller changes to the model. Our method also improved the forecast in RL from sampled returns, in language models of up to 8B parameters, and (with minimal-change training) on the toxic user prompts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
