重新审视浅层对齐假说
Revisiting the Superficial Alignment Hypothesis
AI总结：
本研究通过多系列不同规模大语言模型的实证实验，发现后训练任务性能随微调样本数呈幂律提升、可增强推理与新知识整合能力，指出浅层对齐假说过度简化。
摘要：
浅层对齐假说（Superficial Alignment Hypothesis）认为，语言模型几乎所有的能力与知识都在预训练阶段习得，而后训练阶段仅需为模型赋予正确的风格与格式。我们通过实证研究后训练随微调样本数量增加的缩放行为，并采用客观的任务特定标准化基准进行评估，对上述主张展开重新检验。通过对不同规模的Llama-3、Mistral和Llama-2模型家族开展实验，我们观察到：与预训练缩放定律类似，后训练的任务性能随微调样本数量呈幂律关系提升。这种幂律关系在广泛的能力维度中均成立，包括数学推理、编程、指令遵循和多跳推理。此外，对于数学和多跳推理这类任务，我们发现少量样本仅能在风格上对齐模型，无法使基准测试性能达到饱和。模型性能反而与其推理能力相关，且会随样本数量增加显著提升，这说明除了衡量对人类偏好的对齐程度外，还需要采用客观基准的整体性评估方案。我们还观察到，语言模型并非只能使用预训练阶段习得的知识。通过恰当的后训练，模型在多跳问答等下游任务中整合新知识的能力会大幅提升。综上，这些结果为浅层对齐假说提供了新的认知，表明该假说充其量只是一种过度简化的结论。
英文摘要：
The Superficial Alignment Hypothesis posits that almost all of a language model's abilities and knowledge are learned during pre-training, while post-training is about giving a model the right style and format. We re-examine these claims by empirically studying the scaling behavior of post-training with increasing finetuning examples and evaluating them using objective task-specific standardized benchmarks. Through experiments with the Llama-3, Mistral, and Llama-2 model families of multiple sizes, we observe that, similar to the pre-training scaling laws, post-training task performance scales as a power law against the number of finetuning examples. This power law relationship holds across a broad array of capabilities, including mathematical reasoning, coding, instruction following, and multihop-reasoning. In addition, for tasks like math and multihop reasoning, we observe that a handful of examples merely align the model stylistically but do not saturate performance on the benchmarks. Model performance is instead correlated with its reasoning ability and it improves significantly with more examples, illustrating the need for holistic evaluation programs leveraging objective benchmarks in addition to measurement of alignment to human preferences. We also observe that language models are not necessarily limited to using knowledge learned during pre-training. With appropriate post-training, a model's ability to integrate new knowledge greatly improves on downstream tasks like multihop question-answering. Taken together, these results shed new light on the Superficial Alignment Hypothesis, suggesting that it is, at best, an over-simplification.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
