大语言模型的频谱特征
Spectral Signatures of Large Language Models
AI总结:
针对大规模管理和量化大语言模型的挑战,采用基于重尾自正则化理论的频谱形状指标,以权重经验谱密度形状信息为模型频谱特征,可用于模型谱系追踪等,还构建语料库进行基准测试。
摘要:
快速增长的公开大语言模型库给大规模系统管理和量化带来挑战。任务特定基准不足,为此基于重尾自正则化理论采用频谱形状指标管理和量化大语言模型。该方法用权重经验谱密度形状信息作为模型频谱特征,适用于模型级分析,还进行了基准测试,展示了其在多方面的作用。
英文摘要:
The rapidly growing repository of publicly available large language models (LLMs) presents significant challenges for systematic management and quantification at scale, such as model lineage tracing, licensing, and evaluation. However, task-specific benchmarks are insufficient for this setting, as LLMs differ widely in architectures, scales, and training procedures. To address this challenge, we adopt spectral shape-based metrics for managing and quantifying LLMs based on Heavy-Tailed Self-Regularization theory. Our approach uses the shape information of the weight empirical spectral density as a compact spectral signature of each model. This signature captures intrinsic properties of pretrained models and remains robust during post-training, making it suitable for model-level analysis. In addition, this metric is data-free, computationally-efficient, and scale-invariant, enabling large-scale analysis in practice. Moreover, we curate a large and diverse model corpus consisting of major open-source LLM families, and use it to systematically benchmark spectral and non-spectral metrics across models and downstream tasks. We show that our spectral signature supports the tracking of the model lineage, the unsupervised clustering of similar models, and the quantification of the model performance. Overall, the proposed spectral signature provides a meaningful proxy for broad performance trends across LLMs, enabling efficient organization, comparison, and analysis of large model collections.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。