arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.04413cs.LG
收藏

仅帮助性微调的(错误)泛化

(Mis)generalization of Helpful-Only Fine-tuning

Anthropic Fellows Program(Anthropic 合作者计划);Anthropic

Mohammad Omar Khursheed, Baram Sosis, Fabien Roger

AI总结:

研究仅帮助性训练(不拒绝用户意图)的模型在泛化中的缺陷,发现其存在涌现错位、残余拒绝行为、低可操控性、谄媚和不连贯角色等问题,并提出合成文档微调和添加角色相关问题来缓解。

摘要:

仅帮助性模型,即训练为始终遵循用户意图的模型,对于危险能力评估和AI研发中拒绝行为会成为障碍的其他领域具有价值。关于仅帮助性训练的泛化特性知之甚少:仅帮助性模型比其无害对应模型拒绝更少,但先前工作未研究其对齐的其他维度。我们研究了现有仅帮助性模型的缺陷。我们发现一些模型表现出涌现错位,其他模型存在残余拒绝行为,大多数模型显示出低可操控性、谄媚和不连贯角色。我们表明简单的反拒绝训练可能导致其中许多问题。然而,这些问题并非仅帮助性训练的必要后果:我们证明合成文档微调和向SFT及RL添加角色相关问题可以缓解它们。

英文摘要:

Helpful-only models, that is, models that are trained to always follow user intent, are valuable for dangerous capability evaluations and other areas of AI R&D where refusals would be an obstacle. Little is known about the generalization properties of helpful-only training: helpful-only models refuse less than their harmless counterparts, but previous work has not studied other dimensions of their alignment. We study the shortcomings of existing helpful-only models. We find that some show emergent misalignment, others have residual refusal behaviors, and most show poor steerability, sycophancy, and incoherent character. We show that simple anti-refusal training can cause many of these issues. None of these problems are necessary consequences of helpful-only training, though: we show that synthetic document fine-tuning and adding character-related questions to SFT and RL can mitigate them.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑