层级对齐：通过功能层特化实现大语言模型的外科手术式微调
Hierarchical Alignment: Surgical Fine-Tuning via Functional Layer Specialization in Large Language Models
AI总结：
本文提出层级对齐方法，对LLM不同功能层（局部、中间、全局）分别进行DPO微调，在Llama-3.1-8B和Qwen1.5-7B上验证，全局对齐显著提升事实一致性与逻辑连贯性，且避免标准DPO的对齐税，实现更高效可控的模型对齐。
摘要：
现有的大语言模型（LLM）对齐技术，如直接偏好优化（DPO），通常将模型视为一个整体，对所有层施加统一的优化压力。这种方法忽视了Transformer架构内部的功能特化，不同层已知处理从语法到抽象推理的不同任务。在本文中，我们挑战这种一刀切的范式，引入层级对齐（Hierarchical Alignment），一种新颖的方法，将针对性的DPO应用于模型层的不同功能块：局部（语法）、中间（逻辑）和全局（事实性）。通过对Llama-3.1-8B和Qwen1.5-7B等最先进模型使用LoRA进行外科手术式微调的一系列受控实验，我们的结果由强大的LLM-as-Judge评估，显示出显著且可预测的改进。具体而言，对齐局部层（Local-Align）增强了语法流畅性。更重要的是，对齐全局层（Global-Align）不仅如假设般提高了事实一致性，而且被证明是增强逻辑连贯性的最有效策略，超越了所有基线。关键的是，所有层级策略都成功避免了标准DPO中观察到的“对齐税”，即流畅性提升以逻辑推理能力下降为代价。这些发现为模型对齐建立了一条更资源高效、可控且可解释的路径，凸显了从整体优化转向结构感知的外科手术式微调以构建更先进、更可靠的LLM的巨大潜力。
英文摘要：
Existing alignment techniques for Large Language Models (LLMs), such as Direct Preference Optimization (DPO), typically treat the model as a monolithic entity, applying uniform optimization pressure across all layers. This approach overlooks the functional specialization within the Transformer architecture, where different layers are known to handle distinct tasks from syntax to abstract reasoning. In this paper, we challenge this one-size-fits-all paradigm by introducing Hierarchical Alignment, a novel method that applies targeted DPO to distinct functional blocks of a model's layers: local (syntax), intermediate (logic), and global (factuality). Through a series of controlled experiments on state-of-the-art models like Llama-3.1-8B and Qwen1.5-7B using LoRA for surgical fine-tuning, our results, evaluated by a powerful LLM-as-Judge, demonstrate significant and predictable improvements. Specifically, aligning the local layers (Local-Align) enhances grammatical fluency. More importantly, aligning the global layers (Global-Align) not only improves factual consistency as hypothesized but also proves to be the most effective strategy for enhancing logical coherence, outperforming all baselines. Critically, all hierarchical strategies successfully avoid the "alignment tax" observed in standard DPO, where gains in fluency come at the cost of degraded logical reasoning. These findings establish a more resource-efficient, controllable, and interpretable path for model alignment, highlighting the immense potential of shifting from monolithic optimization to structure-aware surgical fine-tuning to build more advanced and reliable LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
