基础模型规模下的Hessian谱分析
Hessian Spectral Analysis at Foundation Model Scale
AI总结:
本研究在大规模基础模型上实现了Hessian谱的准确分析,揭示了块对角曲率近似在中等规模LLM中的失效问题,展示了谱探测的计算效率与实际应用价值。
摘要:
准确的大型模型Hessian谱一直难以获得,导致大多数先前工作依赖于小型模型或强结构近似。我们展示了在前沿规模上对真实Hessian进行忠实谱分析是可行的。使用与完全分片数据并行兼容的shard局部有限差分Hessian向量积,我们对具有高达1000亿参数的开源语言模型进行了随机Lanczos四次方程,产生出超出子1000亿规模的首个大规模谱密度估计。我们描述了该流程的数值行为,包括有限差分偏差、浮点噪声放大及其对fp32和bf16中Krylov稳定性的影响,并推导出经过实验证明的实用操作范围。我们进一步提供了端到端的运行时间和内存缩放定律,表明完整的运算符谱探测仅在第一阶训练上带来适度的常数因子开销。关键的是,直接访问Hessian揭示了广泛使用的块对角曲率近似可能在中等规模LLM中彻底失效,表现出数量级的相对误差和较差的方向对齐。共同,我们的结果表明,基础模型Hessian谱既可计算,又在现有近似下存在定性偏差,为大规模的曲率分析打开了大门。
英文摘要:
Accurate Hessian spectra of foundation models have remained out of reach, leading most prior work to rely on small models or strong structural approximations. We show that faithful spectral analysis of the true Hessian is tractable at frontier scale. Using shard-local finite-difference Hessian vector products compatible with Fully Sharded Data Parallelism, we perform stochastic Lanczos quadrature on open-source language models with up to 100B parameters, producing the first large-scale spectral density estimates beyond the sub-10B regime. We characterize the numerical behavior of this pipeline, including finite-difference bias, floating-point noise amplification, and their effect on Krylov stability in fp32 and bf16, and derive practical operating regimes that are validated empirically. We further provide end-to-end runtime and memory scaling laws, showing that full-operator spectral probing incurs only a modest constant-factor overhead over first-order training. Crucially, direct access to the Hessian reveals that widely used block-diagonal curvature approximations can fail catastrophically, exhibiting order-one relative error and poor directional alignment even in mid-scale LLMs. Together, our results demonstrate that foundation-model Hessian spectra are both computable and qualitatively misrepresented by prevailing approximations, opening the door to principled curvature-based analysis at scale.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。