预训练课程实现选择性微调
Pretraining Curricula Enable Selective Fine-tuning
AI总结:
研究预训练课程如何影响学习、泛化和微调选择性,对比平衡与不平衡预训练方式,发现不平衡预训练促进上下文学习并提高拒绝微调选择性,还扩展到合成语言学习任务。
摘要:
Transformer遵循隐式课程,某些任务先于其他任务学习。但明确的预训练课程如何影响学习、泛化和微调的选择性尚不清楚。本文比较平衡或不平衡方式预训练任务的课程,表明不平衡学习促进上下文学习并提高拒绝微调的选择性,还扩展到合成语言学习任务,结果表明不平衡预训练课程或为促进解缠表示的重要工具。
英文摘要:
Transformers follow implicit curricula whereby some tasks are learned before others. However, how explicit pretraining curricula influence learning, generalization, and the selectivity of fine-tuning is unclear. This is important for AI safety, where fine-tuning is used to selectively suppress misaligned behaviors. Here, we compare curricula that pretrain tasks in a balanced (sampled uniformly) or an imbalanced (one task early, the other late) fashion. We show that imbalanced learning of two conflicting copy tasks promotes in-context learning and improves the selectivity of refusal fine-tuning. Ablations and activation patching show that this occurs because imbalanced pretraining encourages tasks to be disentangled in separable neural circuits, whereas balanced training routes both tasks through a common pathway. We extend these findings to a synthetic language learning task involving rule-consistent and rule-violating data, where imbalanced curricula similarly lead to more localized, less entangled rule representations, resulting in more robust rule-following behavior. Together, these results suggest that imbalanced pretraining curricula may be an important tool for promoting disentangled representations, with direct consequences for the precision and reliability of safety fine-tuning.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。