DOCS:量化权重相似性以深入理解大型语言模型
DOCS: Quantifying Weight Similarity for Deeper Insights into Large Language Models
AI总结:
本文提出余弦相似性分布(DOCS)指标以量化LLMs权重矩阵相似性,发现相邻层高相似性聚集暗示深度功能特化,并证明其在正交矩阵上的理论有效性,为理解模型架构与开发高效可解释模型提供工具。
摘要:
我们引入了一种新指标——余弦相似性分布(DOCS),用于定量评估大型语言模型(LLMs)中权重矩阵间的相似性,旨在促进对其复杂架构的分析。利用DOCS,我们的分析揭示了最新开源LLMs中令人感兴趣的模式:相邻层通常表现出高权重相似性并倾向于形成簇,表明存在深度方向的功能特化。此外,我们证明DOCS在量化正交矩阵相似性方面具有理论有效性,鉴于LLMs中正交初始化的普遍性,这一点至关重要。本研究有助于更深入理解LLM的架构与行为,所提供的工具对开发更高效、更可解释的模型具有潜在意义。
英文摘要:
We introduce a novel index, the Distribution of Cosine Similarity (DOCS), for quantitatively assessing the similarity between weight matrices in Large Language Models (LLMs), aiming to facilitate the analysis of their complex architectures. Leveraging DOCS, our analysis uncovers intriguing patterns in the latest open-source LLMs: adjacent layers frequently exhibit high weight similarity and tend to form clusters, suggesting depth-wise functional specialization. Additionally, we prove that DOCS is theoretically effective in quantifying similarity for orthogonal matrices, a crucial aspect given the prevalence of orthogonal initializations in LLMs. This research contributes to a deeper understanding of LLM architecture and behavior, offering tools with potential implications for developing more efficient and interpretable models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。