arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.02091cs.AI
收藏

解析大语言模型中层的作用:检索、知识与推理

Demystifying the Roles of LLM Layers in Retrieval, Knowledge, and Reasoning

Emory University(埃默里大学);University of Tuebingen(图宾根大学);University of Surrey(萨里大学);Max Planck Institute for Intelligent Systems(智能系统马克斯·普朗克研究所);ELLIS Institute Tübingen(图宾根ELLIS研究所)

Xinyuan Song, Keyu Wang, PengXiang Li, Lu Yin, Shiwei Liu

AI总结:

本研究解析了大语言模型中不同深度层在检索、知识和推理中的作用,发现浅层主导知识和检索,深层影响推理,且不同评估方法下深度利用存在显著差异。

摘要:

近期研究表明,大语言模型(LLMs)的深层结构对表示学习贡献有限,可以去除而不影响性能。然而,这些结论通常来自狭窄的评估,可能忽略了模型行为的重要方面。在本工作中,我们系统地研究了深度在多种维度上的利用情况,包括评估协议、任务类别和模型架构。我们的分析证实,非常深层的层通常不如早期层有效,但其贡献会显著依赖于评估设置。在基于似然的度量下没有生成时,修剪大多数层能保持性能,只有初始几层是关键。相比之下,基于生成的评估揭示了中层和深层层在支持推理和维持长程连贯性中的不可或缺作用。我们进一步发现,知识和检索集中在浅层组件,而推理准确性严重依赖深层层--但可以通过蒸馏重塑。这些结果表明,LLM中的深度使用高度异质且依赖上下文,强调了在解释和压缩大型模型时需要任务、度量和模型意识的视角。

英文摘要:

Recent studies suggest that the deeper layers of Large Language Models (LLMs) contribute little to representation learning and can often be removed without significant performance loss. However, such claims are typically drawn from narrow evaluations and may overlook important aspects of model behavior. In this work, we present a systematic study of depth utilization across diverse dimensions, including evaluation protocols, task categories, and model architectures. Our analysis confirms that very deep layers are generally less effective than earlier ones, but their contributions vary substantially with the evaluation setting. Under likelihood-based metrics without generation, pruning most layers preserves performance, with only the initial few being critical. By contrast, generation-based evaluation uncovers indispensable roles for middle and deeper layers in enabling reasoning and maintaining long-range coherence. We further find that knowledge and retrieval are concentrated in shallow components, whereas reasoning accuracy relies heavily on deeper layers -- yet can be reshaped through distillation. These results highlight that depth usage in LLMs is highly heterogeneous and context-dependent, underscoring the need for task-, metric-, and model-aware perspectives in both interpreting and compressing large models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑