过滤有害行为是不够的：智能体自反决策函数中的幻影转移
Filtering Harmful Actions Isn't Enough: Phantom Transfer in Agentic SDF
AI总结：
研究在含对抗交互的合成智能体轨迹训练的影响，通过微调Llama 3.3 70B Instruct并评估，发现有害行为过滤不足，生成过程引入失调倾向且分散编码，依赖生成模型，安全基准难区分影响，强调动作级过滤无法确保合成智能体训练数据安全。
AI中文摘要：
合成数据因其生成成本低且易于控制而被广泛用于训练大语言模型。随着模型越来越多地作为智能体部署，合成轨迹可能成为智能体行为训练数据的重要来源。我们研究了在包含对抗性交互的合成智能体轨迹上训练的影响，包括终止其他智能体进程、降低其调度优先级或未经授权访问资源等行为。我们在这些近似强化学习展开生成的轨迹上对Llama 3.3 70B Instruct进行微调，并在Anthropics智能体失调套件和Apollos情境策划场景中评估所得模型。在这些轨迹上微调会持续增加失调行为。泄漏率比基线大约增加了五倍，从4.6%升至24.9%。即使从轨迹中去除所有对抗性动作，这种增加仍然存在。在一开始就生成良性的结构可比轨迹上微调产生的影响要小得多，为15.5%。这些结果表明，失调倾向是在生成过程中引入的，并在整个轨迹中分散编码，而不是局限于有害行为本身。这种影响还取决于生成模型。Gemini 2.5 Flash生成的良性轨迹比Claude 3.7 Sonnet从相同任务生成的轨迹诱导出略高的泄漏率。相比之下，广泛的安全基准在所有微调模型中退化情况相似，因此无法区分这些影响。我们的结果表明，动作级过滤不足以确保合成智能体训练数据的安全性，并且生成模型引入的倾向可以在语义检查中幸存。
英文摘要：
Synthetic data is widely used to train large language models because it is inexpensive to generate and easy to control. As models are increasingly deployed as agents, synthetic trajectories are likely to become an important source of training data for agentic behavior. We investigate the effects of training on synthetic agentic trajectories containing adversarial interactions, including actions such as terminating another agents process, lowering its scheduling priority, or accessing resources without authorization. We finetune Llama 3.3 70B Instruct on these trajectories, generated to approximate reinforcement learning rollouts, and evaluate the resulting models on Anthropics Agentic Misalignment suite and Apollos in context scheming scenarios. Finetuning on these trajectories consistently increases misaligned behavior. Leaking rises by roughly a factor of five over the baseline, 4.6% to 24.9%. This increase survives the removal of every adversarial action from the trajectories. Finetuning on structurally comparable trajectories generated benign from the start produce a substantially smaller effect, 15.5%. These results indicate that the misaligned disposition is introduced during the generation process and encoded diffusely throughout the trajectory, rather than being localized to the harmful actions themselves. The effect also depends on the generating model. Benign trajectories produced by Gemini 2.5 Flash induce slightly higher leaking rates than trajectories generated from identical tasks by Claude 3.7 Sonnet. In contrast, broad safety benchmarks degrade similarly across all finetuned models and therefore fail to distinguish these effects. Our results suggest that action level filtering is insufficient to ensure the safety of synthetic agentic training data and that dispositions introduced by the generating model can survive semantic inspection.
