仅前向传播的领域适配（无需跨层反向传播）
Forward Pass Domain Adaptation (Without Cross-Layer Backpropagation)
浏览论文内容
中文总结 AI 辅助
该研究提出仅前向传播的MLP训练（FPO）方法，无需跨层反向传播即可适配大语言模型，提升吞吐量、降低内存开销，在保持域外基准性能的同时优化域内困惑度，且耗时低于限定SFT的方案。
中文摘要 AI 辅助
仅前向传播的MLP训练（FPO）可在不通过模型主体进行反向传播的情况下适配大语言模型，实现了标准微调2.7至3.2倍的吞吐量，同时峰值训练内存减少约40%，且使域外基准保持在基线的种子噪声范围内，这是全网络微调无法可靠复现的特性。FPO基于一项经验观察：在Transformer的后期层中，我们调研的6个公开模型的输出层预测误差与真实梯度的余弦相似度为0.47至0.59。我们引入了一项两分钟诊断方法，可量化任意模型各层的这种近似程度，从而确定后期层适配的可行性区域。基于该诊断，FPO在输出层计算单一误差信号，并将其应用于每个目标层，层间无信号传播，且全程不构建自动微分图。我们在三个模型系列（OLMo-2-7B、Qwen3-8B、Falcon3-7B）上对FPO进行评估，在所有三个模型上，FPO均实现了域内困惑度的提升，且使MMLU、ARC-Challenge、HellaSwag和Winogrande保持在基线的种子噪声范围内。将SFT限定在FPO的目标层以进入该状态也是可行的，但耗时成本为FPO的2.2倍。
英文摘要
Forward-Pass-Only MLP training (FPO) adapts large language models without a backward pass through the model body, achieving 2.7--3.2x the throughput of standard fine-tuning at ~40% less peak training memory, while leaving off-domain benchmarks within seed-noise of baseline, a property that full-network fine-tuning does not reliably reproduce. FPO rests on a single empirical observation: at late layers of a transformer, the output-layer prediction error approximates the true gradient with cosine similarity 0.47--0.59 across six public models we survey. We introduce a two-minute diagnostic that quantifies this approximation per layer for any model, identifying where late-layer adaptation is viable. Informed by the diagnostic, FPO computes a single error signal at the output and applies it to each target layer. No signal is propagated between layers, and no autograd graph is constructed at any point. We evaluate FPO on three model families (OLMo-2-7B, Qwen3-8B, Falcon3-7B). Across all three, FPO produces in-domain perplexity improvement and leaves MMLU, ARC-Challenge, HellaSwag, and Winogrande within seed-noise of baseline. Localizing SFT to FPO's target layers to enter this regime is also feasible, but at 2.2x the wall-clock cost of FPO.
发表机构
- University of California, Santa Cruz（加州大学圣克鲁兹分校）
- University of Melbourne（墨尔本大学）
机构由 AI 辅助整理，请以论文原文为准。
