arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.01569cs.AIcs.CL
收藏

InvThink:用于更安全语言模型的预mortem推理

InvThink: Premortem Reasoning for Safer Language Models

MIT(麻省理工学院);Google Research(谷歌研究院);Google DeepMind(谷歌DeepMind);Samsung Research(三星研究)

Yubin Kim, Taehan Kim, Eugene Park, Chunjong Park, Cynthia Breazeal, Daniel McDuff, Hae Won Park

AI总结:

InvThink通过预mortem推理框架提升语言模型安全性,通过枚举、分析和约束潜在故障来生成响应,相比现有方法在安全性和减少有害行为方面表现更优。

摘要:

我们提出了InvThink,一种训练和提示框架,要求模型在生成最终响应前枚举、分析并约束潜在故障。与仅优化安全最终响应的传统安全对齐方法不同,InvThink将生成过程分为三个步骤:(1) 枚举潜在危害,(2) 分析其后果,(3) 在显式缓解约束下生成响应。我们观察到三个发现:(i) 相比现有安全提示和对齐基线,InvThink在更大模型规模上表现出更高的安全性评分。(ii) InvThink减轻了安全税。使用InvThink训练的模型在标准基准上保持推理能力。(iii) 除了通用安全任务外,InvThink还减少了专业伦理领域(医学、金融、法律)和代理不一致场景中的有害行为,相较于零样本基线,有害性减少了高达32%,相较于SafetyPrompt减少了16%。我们通过监督微调和基于GRPO的强化学习扩展了InvThink,应用于三个LLM家族。

英文摘要:

We present InvThink, a training and prompting framework that requires the model to enumerate, analyze, and constrain potential failures before generating its final response. Unlike existing safety alignment methods that optimize only for safe final responses, InvThink structures generation into three steps: (1) enumerate potential harms, (2) analyze their consequences, (3) generate the response under explicit mitigation constraints. We observe three findings: (i) InvThink shows higher safety scores at larger model sizes, compared to existing safety prompting and alignment baselines. (ii) InvThink mitigates the safety tax. Models trained with INVTHINK preserve their reasoning capability on standard benchmarks. (iii) beyond general safety tasks, InvThink also reduces harmful behavior in professional ethics domains (medicine, finance, law) and in agentic misalignment scenarios, achieving up to 32% reduction in harmfulness over zero-shot baselines and 16% over SafetyPrompt. We extend InvThink with supervised fine-tuning, and GRPO-based reinforcement learning across three LLM families.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑