Diffract：大语言模型领域自适应的谱视角
Diffract: Spectral View of LLM Domain Adaptation
AI总结：
本研究提出Diffract工具包，通过分析持续预训练的大语言模型的权重矩阵奇异值谱，发现可选择性回退低重要性注意力头以提升领域适配性能，实现高效的领域自适应。
AI中文摘要：
我们研究持续预训练（CPT）作为将通用大语言模型适配到数学、指令、代码和自然文本等专业领域的机制。通过对权重矩阵进行奇异值分解，我们发现CPT使奇异值谱基本保持不变，适配主要由奇异向量的变化驱动。对注意力头投影矩阵的分析显示出强烈的、依赖领域的头异质性，我们利用这一点定义了头重要性准则：最多可移除60%的头更新而不会出现可测量的质量损失。与完全训练的基线相比，选择性地将低重要性头回退到预训练状态可将基准准确率提升多达4%。最后，我们确定了领域连通性——在CPT检查点之间的线性插值可产生平滑的领域质量插值，且在任一领域均无明显性能下降——并发布了Diffract，这是一个用于对数十亿参数模型进行可扩展谱分析的开源工具包。
英文摘要：
We study continual pre-training (CPT) as a mechanism for adapting general-purpose large language models to specialized domains: mathematics, instruction, code, and natural text. Using singular value decomposition of weight matrices, we find that CPT leaves singular value spectra largely invariant, with adaptation driven mainly by changes in singular vectors. An analysis of attention-head projection matrices reveals strong, domain-dependent head heterogeneity, which we exploit to define a head importance criterion: up to 60% of head updates can be removed without measurable quality loss. Selectively rewinding low-importance heads to their pre-trained state improves benchmark accuracy by up to 4% versus the fully trained baseline. Finally, we identify domain connectivity - linear interpolation between CPT checkpoints yields smooth domain-quality interpolation without notable degradation on either domain - and release Diffract, an open-source toolkit for scalable spectral analysis of billion-parameter models.
