目标至关重要：微调目标影响安全、鲁棒性和人格漂移
Objective Matters: Fine-Tuning Objectives Shape Safety, Robustness, and Persona Drift
AI总结：
本文研究了微调目标对LLM安全性和鲁棒性的影响，发现目标选择在不同训练规模下对安全、鲁棒性和人格漂移有显著影响。
摘要：
对LLM进行良性数据的微调仍可能降低对齐性和对抗性鲁棒性，但直接分析微调目标在塑造这些安全结果中的作用仍有限。我们展示了六个微调目标——监督微调、直接偏好优化、条件微调、接种提示、比值偏好优化和KL正则化微调——在保持数据、领域、架构和优化固定的情况下进行受控比较。在闭合形式推理和开放性生成任务中，我们发现目标选择会在安全-能力前沿上引起系统性的、规模依赖的转移。在小规模训练预算下，鲁棒性在不同目标之间相似，但能力差异明显。在较大预算下，目标差异显著：监督和偏好基于的微调紧密将能力提升与增加对抗性脆弱性和人格漂移联系起来，而限制学习信号的目标——尤其是ORPO和KL正则化——显著减轻了这两种影响。因此，微调目标在小规模下对安全性影响较小，但随着训练规模的增加，它成为对抗性鲁棒性和潜在人格稳定性的主要驱动因素。
英文摘要：
Fine-tuning LLMs on benign data can still degrade alignment and adversarial robustness, yet direct analysis of the role of fine-tuning objectives in shaping these safety outcomes remain limited. We present a controlled comparison of six fine-tuning objectives -- Supervised Fine-Tuning, Direct Preference Optimization, Conditional Fine-Tuning, Inoculation Prompting, Odds Ratio Preference Optimization, and KL-regularized fine-tuning -- holding data, domain, architecture, and optimization fixed. Across closed-form reasoning and open-ended generation tasks, we find that objective choice induces systematic, scale-dependent shifts along the safety-capability frontier. At small training budgets, robustness is similar across objectives but capability differs. At larger budgets, objectives diverge sharply: supervised and preference-based tuning tightly couple capability gains to increased adversarial vulnerability and persona drift, while objectives that constrain learning signals -- especially ORPO and KL-regularization -- substantially mitigate both. Fine-tuning objectives therefore matter little for safety at small scales but become a primary driver of adversarial robustness and latent persona stability as training scale increases.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
