接种提示(Inoculation Prompting):在训练时指示大语言模型做出不当行为可提升测试时的对齐效果
Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment
AI总结:
针对大语言模型在有缺陷监督信号下训练会习得奖励黑客等不良行为的问题,提出反直觉的接种提示技术,通过训练时主动要求模型做出不良行为来阻止其习得该行为,在四种设置中验证了其有效性且不损害所需能力。
摘要:
大语言模型有时会在存在缺陷的监督信号下训练,导致奖励黑客行为、谄媚等不良行为的出现。提升监督质量成本高昂甚至不可行,因此亟需在训练信号存在缺陷的情况下仍能优化模型习得行为的方法。我们提出Inoculation Prompting(IP,接种提示),这是一种简单却反直觉的技术,通过修改训练提示明确要求模型做出不良行为,来阻止模型习得该行为。例如,为了对奖励黑客行为进行接种,我们修改监督微调中使用的提示,要求生成仅能在给定测试用例上运行、但在其他输入上失效的代码。在四种实验设置中,我们发现IP减少了模型对不良行为的习得,同时不会大幅削弱对所需能力的学习。我们还表明,在微调前能更强地诱发不良行为的提示,在训练阶段使用时能更有效地对该行为进行接种;这可作为筛选有效接种提示的启发式方法。总体而言,IP是一种简单有效的方法,可控制模型从微调中泛化的方式,在不显著干扰所需能力的前提下阻止不良行为的习得。
英文摘要:
Large language models are sometimes trained with imperfect oversight signals, leading to undesired behaviors such as reward hacking and sycophancy. Improving oversight quality can be expensive or infeasible, motivating methods that improve learned behavior despite an imperfect training signal. We introduce Inoculation Prompting (IP), a simple but counterintuitive technique that prevents learning of an undesired behavior by modifying training prompts to explicitly request it. For example, to inoculate against reward hacking, we modify the prompts used in supervised fine-tuning to request code that only works on provided test cases but fails on other inputs. Across four settings we find that IP reduces the learning of undesired behavior without substantially reducing the learning of desired capabilities. We also show that prompts which more strongly elicit the undesired behavior prior to fine-tuning more effectively inoculate against the behavior when used during training; this serves as a heuristic to identify promising inoculation prompts. Overall, IP is a simple yet effective way to control how models generalize from fine-tuning, preventing learning of undesired behaviors without substantially disrupting desired capabilities.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。