arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.14381cs.CLcs.LG
收藏

探究大型语言模型中的层重要性

Investigating Layer Importance in Large Language Models

National University of Singapore(新加坡国立大学);PayPal(贝宝)

Yang Zhang, Yanfei Dong, Kenji Kawaguchi

AI总结:

本研究通过Shapley值和层消融实验评估LLMs中各层的重要性,发现某些早期“基石层”对模型性能起主导作用,移除会导致性能崩溃退化为随机猜测,移除非基石层则影响甚微。

摘要:

大型语言模型(LLMs)因其理解和处理文本的卓越能力而备受关注。然而,LLMs在很大程度上仍然是不透明的。对LLMs缺乏理解阻碍了其在安全关键场景中的部署,并妨碍了更好模型的开发。在本研究中,我们通过探究LLMs中各单独层的显著性来加深对LLM的理解。我们提出了一种高效的采样方法,利用Shapley值(特征归因和数据评估中广泛使用的解释框架)来忠实地评估各层的重要性。此外,我们进行了层消融实验,以评估排除特定层所导致的性能下降。我们的发现揭示了基石层的存在,其中某些早期层对其他层展现出主导性贡献。移除一个基石层会导致模型性能的急剧崩溃,通常使其退化为随机猜测。相反,移除非基石层仅导致边际的性能变化。本研究识别了LLMs中的基石层,并强调了它们在未来研究中的关键作用。

英文摘要:

Large language models (LLMs) have gained increasing attention due to their prominent ability to understand and process texts. Nevertheless, LLMs largely remain opaque. The lack of understanding of LLMs has obstructed the deployment in safety-critical scenarios and hindered the development of better models. In this study, we advance the understanding of LLM by investigating the significance of individual layers in LLMs. We propose an efficient sampling method to faithfully evaluate the importance of layers using Shapley values, a widely used explanation framework in feature attribution and data valuation. In addition, we conduct layer ablation experiments to assess the performance degradation resulting from the exclusion of specific layers. Our findings reveal the existence of cornerstone layers, wherein certain early layers can exhibit a dominant contribution over others. Removing one cornerstone layer leads to a drastic collapse of the model performance, often reducing it to random guessing. Conversely, removing non-cornerstone layers results in only marginal performance changes. This study identifies cornerstone layers in LLMs and underscores their critical role for future research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑