绘制LLM微调的实证定律以用于科学多学科学习
Charting Empirical Laws for LLM Fine-Tuning in Scientific Multi-Discipline Learning
AI总结：
本研究通过构建五学科语料库，揭示了多学科LLM微调的四个实证定律，为科学领域通用LLM的发展提供了可操作的指导。
摘要：
尽管大型语言模型（LLMs）通过在单个科学领域内进行微调已取得了强劲的表现，但其在跨学科环境中的学习动态仍缺乏深入理解，尽管通过跨领域知识协同可以带来更广泛的泛化能力和应用范围。在本工作中，我们提出了对多学科LLM微调的首次系统研究，构建了一个五学科语料库，并分析了全微调、LoRA、LoRA-MoE以及LoRA组合的学习模式。特别地，我们的研究表明，多学科学习比单学科训练更具变异性，并提炼出四个一致的实证定律：（1）平衡后再多样化：低资源学科在未通过多样性意识上采样进行缓解的情况下会导致性能下降；（2）合并后再对齐：恢复指令遵循能力对于跨学科协同至关重要；（3）优化后再扩展：参数扩展在没有先前设计优化的情况下提供有限的收益；（4）共享后再专业化：不对称的LoRA-MoE通过共享的低秩投影在极小的可训练参数下产生稳健的收益。这些定律共同构成了一个实用的原理性多学科微调配方，并为开发具有泛化能力的科学LLM提供了可操作的指导。
英文摘要：
While large language models (LLMs) have achieved strong performance through fine-tuning within individual scientific domains, their learning dynamics in multi-disciplinary contexts remains poorly understood, despite the promise of improved generalization and broader applicability through cross-domain knowledge synergy. In this work, we present the first systematic study of multi-disciplinary LLM fine-tuning, constructing a five-discipline corpus and analyzing learning patterns of full fine-tuning, LoRA, LoRA-MoE, and LoRA compositions. Particularly, our study shows that multi-disciplinary learning is substantially more variable than single-discipline training and distills four consistent empirical laws: (1) Balance-then-Diversity: low-resource disciplines degrade performance unless mitigated via diversity-aware upsampling; (2) Merge-then-Align: restoring instruction-following ability is critical for cross-discipline synergy; (3) Optimize-then-Scale: parameter scaling offers limited gains without prior design optimization; and (4) Share-then-Specialize: asymmetric LoRA-MoE yields robust gains with minimal trainable parameters via shared low-rank projection. Together, these laws form a practical recipe for principled multi-discipline fine-tuning and provide actionable guidance for developing generalizable scientific LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
