arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.00495cs.CLcs.AI
收藏

基于合成偏好数据的语言模型可配置安全调优

Configurable Safety Tuning of Language Models with Synthetic Preference Data

Komorebi AI

Victor Gallego

AI总结:

本文提出可配置安全调优(CST)方法,通过合成偏好数据增强DPO并引入系统提示,使大语言模型在推理阶段可灵活启用或禁用安全偏好,实验证明其能保持模型原有功能,适用于可配置部署。

摘要:

当前最先进的语言模型微调技术,如直接偏好优化(DPO),通过将预定义行为硬编码到模型中,限制了用户控制能力。为解决这一问题,我们提出了一种新方法——可配置安全调优(CST),该方法利用合成偏好数据对DPO进行增强,从而在推理阶段实现大语言模型(LLM)的灵活安全配置。CST通过引入一个指定安全配置的系统提示,克服了普通DPO的局限性,使LLM部署者能够根据自身需求禁用或启用安全偏好,只需更改系统提示即可。我们的实验评估表明,CST能够成功管理不同的安全配置,同时保持LLM的原有功能,证明它是一种适用于可配置部署的稳健方法。数据和模型可在 https://github.com/vicgalle/configurable-safety-tuning 获取。

英文摘要:

State-of-the-art language model fine-tuning techniques, such as Direct Preference Optimization (DPO), restrict user control by hard-coding predefined behaviors into the model. To address this, we propose a novel method, Configurable Safety Tuning (CST), that augments DPO using synthetic preference data to facilitate flexible safety configuration of LLMs at inference time. CST overcomes the constraints of vanilla DPO by introducing a system prompt specifying safety configurations, enabling LLM deployers to disable/enable safety preferences based on their need, just changing the system prompt. Our experimental evaluations indicate that CST successfully manages different safety configurations and retains the original functionality of LLMs, showing it is a robust method for configurable deployment. Data and models available at https://github.com/vicgalle/configurable-safety-tuning

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑