帮助有害:后训练中领域依赖的中期训练同情价值观退化
Helpfulness Hurts: Domain-Dependent Degradation of Mid-Trained Compassion Values Under Post-Training
AI总结:
研究后训练数据领域对中期训练同情价值观的影响,发现帮助性训练比编程训练更显著降低动物同情和道德推理能力,但跨语言迁移存在差异。
摘要:
标准后训练流程应用监督微调(SFT)和强化学习(RL)使语言模型具有帮助性,但这些过程可能无意中削弱预训练期间灌输的价值观。我们研究了后训练数据的领域是否差异性地影响在同情导向合成数据上中期训练的Llama 3.1 8B模型中动物同情价值观的保留,使用SFT(通过Dolly-15k的帮助性 vs. 通过Magicoder-110K的编程)和GRPO(通过RLHFlow的帮助性 vs. 通过Magicoder的编程),并在动物伤害基准(AHB 2.2)和MORU基准(不确定性下的道德推理)上评估。在AHB上,帮助性训练相对于编程训练显著降低动物同情(SFT:35.7% vs. 65.2%;GRPO:18.7% vs. 32.0%),这一结果在两个独立的帮助性数据集和两个训练范式上重复。在英文MORU项目上,帮助性训练将一般道德推理降低25.5个百分点(46.4% vs. 71.9%),这一显著差距在幅度上与同情效应相当。然而,这种效应不会跨语言迁移:在多语言MORU基准上,领域效应消失(SFT:52.3% vs. 51.2%)。相比之下,动物同情效应在语言间一致迁移,Magicoder在AHB上相对于基础模型的百分点增益在非英文项目上是英文项目的4.5倍。这种分歧表明,通过中期训练灌输的价值观比领域特定后训练带来的推理改进编码得更深且更具跨语言性。这些结果表明,对于在承载价值观的中期训练基础上构建的实验室,编程领域后训练可能比帮助性后训练更好地保留中期训练的价值观,同时不损害一般推理能力。
英文摘要:
Standard post-training pipelines apply supervised fine-tuning (SFT) and reinforcement learning (RL) to make language models helpful, but these processes may inadvertently degrade values instilled during pre-training. We investigate whether the domain of post-training data differentially affects the retention of animal compassion values in a Llama 3.1 8B model mid-trained on compassion-oriented synthetic data, using both SFT (helpfulness via Dolly-15k vs. coding via Magicoder-110K) and GRPO (helpfulness via RLHFlow vs. coding via Magicoder), evaluated on the ANIMA 2.2 benchmark and MORU benchmark (Moral Reasoning Under Uncertainty). Helpfulness training significantly degrades animal compassion relative to coding training on ANIMA (SFT: 35.7% vs. 65.2%; GRPO: 18.7% vs. 32.0%), replicating across two independent helpfulness datasets and two training paradigms. On English MORU items, helpfulness training degrades general moral reasoning by 25.5 percentage points (46.4% vs. 71.9%), a striking gap that rivals the compassion effect in magnitude. However, this effect does not transfer cross-lingually: on the multilingual MORU benchmark, the domain effect disappears (SFT: 52.3% vs. 51.2%). In contrast, the animal compassion effect transfers consistently across languages, with Magicoder's ANIMA percentage-point gain over the base model 4.5 times larger on non-English items than English items. This divergence suggests that values instilled through mid-training are encoded more deeply and cross-lingually than reasoning improvements from domain-specific post-training. These results suggest that, for labs building on value-laden mid-training, coding-domain post-training may better preserve mid-trained values than helpfulness post-training without harming general reasoning capabilities.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。