arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.00537cs.LGcs.ITmath.IT
收藏

语言模型中的谱缩放定律:前馈网络利用其潜在空间的效率如何?

Spectral Scaling Laws in Language Models: How Effectively Do Feed-Forward Networks Use Their Latent Space?

New York University(纽约大学)

Nandan Kumar Jha, Brandon Reagen

AI总结:

该研究将前馈网络宽度选择转化为谱利用问题,通过轻量诊断工具发现LLM中FFN存在非对称谱缩放定律,为推理高效的大语言模型设计提供指导。

摘要:

随着大语言模型(LLM)不断缩放,问题不仅在于它们变得多大,还在于其容量被有效利用的程度。现有缩放定律将模型规模与损失关联起来,却忽略了各组件如何利用其潜在空间。我们研究前馈网络(FFN),并将宽度选择重新定义为一个谱利用问题。我们使用一套轻量级诊断工具——硬秩(参与比)、软秩(香农秩)、谱集中度以及综合的谱利用指数(SUI),量化了LLaMA、GPT-2和nGPT系列模型中有多少潜在方向被有效激活。我们的核心发现是一条非对称谱缩放定律:软秩与FFN宽度几乎完美地遵循幂律关系,而硬秩仅呈亚线性增长且方差较高。这种非对称性表明,拓宽FFN主要增加了低能量尾部方向,而主导模子空间则很早就饱和了。此外,在更大宽度下,方差会进一步坍缩到一个狭窄的子空间中,导致大部分潜在空间未被充分利用。这些结果将FFN宽度选择重新定义为尾部容量与主导模容量之间的原则性权衡,为推理高效的LLM设计提供了具体指导。

英文摘要:

As large language models (LLMs) scale, the question is not only how large they become, but how much of their capacity is effectively utilized. Existing scaling laws relate model size to loss, yet overlook how components exploit their latent space. We study feed-forward networks (FFNs) and recast width selection as a spectral utilization problem. Using a lightweight diagnostic suite -- Hard Rank (participation ratio), Soft Rank (Shannon rank), Spectral Concentration, and the composite Spectral Utilization Index (SUI) -- we quantify how many latent directions are meaningfully activated across LLaMA, GPT-2, and nGPT families. Our key finding is an asymmetric spectral scaling law: soft rank follows an almost perfect power law with FFN width, while hard rank grows only sublinearly and with high variance. This asymmetry suggests that widening FFNs mostly adds low-energy tail directions, while dominant-mode subspaces saturate early. Moreover, at larger widths, variance further collapses into a narrow subspace, leaving much of the latent space under-utilized. These results recast FFN width selection as a principled trade-off between tail capacity and dominant-mode capacity, offering concrete guidance for inference-efficient LLM design.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑