arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.10288cs.CLcs.LG
收藏

照我这样做(安全地):缓解大型语言模型中任务特定微调的风险

Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models

University of Oxford(牛津大学);Dynamo AI(戴纳摩人工智能公司)

Francisco Eiras, Aleksandar Petrov, Philip H. S. Torr, M. Pawan Kumar, Adel Bibi

AI总结:

本研究揭示任务特定微调可被恶意操纵以诱发危险行为,并提出混合格式匹配的安全数据来高效恢复安全对齐,同时保持任务性能。

摘要:

近期研究表明,在良性指令遵循数据上进行微调可能会无意中破坏安全对齐过程,并增加模型遵从有害查询的倾向。虽然指令遵循微调很重要,但任务特定微调——即模型在具有明确正确答案(如多项选择题)的数据集上进行训练——可以提升模型在专门下游任务上的性能。由于数据结构的差异,理解并缓解任务特定场景中的安全风险与指令遵循情境有所不同。我们的工作展示了恶意行为者如何巧妙地操纵几乎任何任务特定数据集的结构,以诱发显著更危险的模型行为,同时保持表面上的无害性和合理的下游任务性能。为解决此问题,我们提出了一种新颖的缓解策略,即混合入模仿用户数据的任务格式和提示风格的安全数据,证明该方法在重新建立安全对齐的同时保持相似的任务性能方面,比现有基线方法显著更有效且更高效。

英文摘要:

Recent research shows that fine-tuning on benign instruction-following data can inadvertently undo the safety alignment process and increase a model's propensity to comply with harmful queries. While instruction-following fine-tuning is important, task-specific fine-tuning - where models are trained on datasets with clear ground truth answers (e.g., multiple choice questions) - can enhance model performance on specialized downstream tasks. Understanding and mitigating safety risks in the task-specific setting remains distinct from the instruction-following context due to structural differences in the data. Our work demonstrates how malicious actors can subtly manipulate the structure of almost any task-specific dataset to foster significantly more dangerous model behaviors, while maintaining an appearance of innocuity and reasonable downstream task performance. To address this issue, we propose a novel mitigation strategy that mixes in safety data which mimics the task format and prompting style of the user data, showing this is significantly more effective and efficient than existing baselines at re-establishing safety alignment while maintaining similar task performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑