弥散损失抵消嵌入凝结并提升小语言模型的泛化能力
Dispersion Loss Counteracts Embedding Condensation and Improves Generalization in Small Language Models
AI总结：
本研究提出弥散损失以缓解小语言模型中的嵌入凝结现象，通过训练提升模型泛化能力，无需增加参数。
摘要：
大型语言模型（LLMs）通过不断增长的参数数量实现卓越性能，但扩展会带来陡峭的计算成本。为了更好地理解LLM扩展，我们研究了LLM与其较小版本之间的表征差异，旨在复制较大模型的表征质量。我们观察到一种几何现象，我们称之为嵌入凝结，其中某些语言模型的token嵌入会坍缩成一个狭窄的锥形子空间。通过系统分析多个Transformer家族，我们发现小型模型如GPT2和Qwen3-0.6B表现出严重的凝结，而较大的模型如GPT2-xl和Qwen3-32B则更不容易受到这种现象的影响。此外的观察表明，从较大模型的知识蒸馏并不能可靠地缓解嵌入凝结。为对抗这一现象，我们提出了一个弥散损失，该损失在训练期间显式鼓励嵌入弥散。实验表明，它减轻了凝结，恢复了在较大模型中观察到的弥散模式，并在10个基准测试中带来了性能提升。我们相信这项工作为在不增加额外参数的情况下改进小型Transformer提供了一条原则性的路径。
英文摘要：
Large language models (LLMs) achieve remarkable performance through ever-increasing parameter counts, but scaling incurs steep computational costs. To better understand LLM scaling, we study representational differences between LLMs and their smaller counterparts, with the goal of replicating the representational qualities of larger models in the smaller models. We observe a geometric phenomenon which we term $\textbf{embedding condensation}$, where token embeddings collapse into a narrow cone-like subspace in some language models. Through systematic analyses across multiple Transformer families, we show that small models such as $\texttt{GPT2}$ and $\texttt{Qwen3-0.6B}$ exhibit severe condensation, whereas the larger models such as $\texttt{GPT2-xl}$ and $\texttt{Qwen3-32B}$ are more resistant to this phenomenon. Additional observations show that embedding condensation is not reliably mitigated by knowledge distillation from larger models. To fight against it, we formulate a dispersion loss that explicitly encourages embedding dispersion during training. Experiments demonstrate that it mitigates condensation, recovers dispersion patterns seen in larger models, and yields performance gains across 10 benchmarks. We believe this work offers a principled path toward improving smaller Transformers without additional parameters.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
