arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.09563cs.LGcs.CL
收藏

表征重要吗?探索大语言模型的中间层

Does Representation Matter? Exploring Intermediate Layers in Large Language Models

Mila(米拉研究所);New York University(纽约大学);Meta FAIR;University of Kentucky(肯塔基大学)

Oscar Skean, Md Rifat Arefin, Yann LeCun, Ravid Shwartz-Ziv

AI总结:

该研究探索大语言模型中间层表征质量,发现中间层表征对下游任务更具信息量,通过多指标实证揭示架构差异、训练演化及影响因素,为模型架构优化和训练提供指导。

摘要:

理解大语言模型(LLM)中优质表征的定义,对理论认知和实际应用都具有基础意义。本文研究了Transformer、状态空间模型(SSM)等多种LLM架构的中间表征质量,发现中间层往往能为下游任务提供比最终层更具信息量的表征。为衡量表征质量,我们调整并应用了一套最初在其他领域提出的指标,包括提示熵、曲率和增强不变性。实证研究揭示了架构间的显著差异、表征在训练过程中的演化规律,以及输入随机性、提示长度等因素对各层的影响。值得注意的是,我们在部分中间层的熵值中观察到双峰模式,并探讨了其与训练数据相关的潜在解释。总体而言,研究结果阐明了LLM的内部机制,可为架构优化和训练策略提供指导。

英文摘要:

Understanding what defines a good representation in large language models (LLMs) is fundamental to both theoretical understanding and practical applications. In this paper, we investigate the quality of intermediate representations in various LLM architectures, including Transformers and State Space Models (SSMs). We find that intermediate layers often yield more informative representations for downstream tasks than the final layers. To measure the representation quality, we adapt and apply a suite of metrics - such as prompt entropy, curvature, and augmentation-invariance - originally proposed in other contexts. Our empirical study reveals significant architectural differences, how representations evolve throughout training, and how factors like input randomness and prompt length affect each layer. Notably, we observe a bimodal pattern in the entropy of some intermediate layers and consider potential explanations tied to training data. Overall, our results illuminate the internal mechanics of LLMs and guide strategies for architectural optimization and training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑