越远离分布,表示越稀疏:分析LLM中的OOD机制
Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs
AI总结:
研究LLM在面对难度增加的输入时内部表示的适应机制,发现任务难度提升导致最后隐藏层表示稀疏化,提出基于稀疏性的课程学习策略提升性能。
摘要:
本文研究了大型语言模型(LLM)在遇到难度增加的输入时如何调整其内部表示,通过量化分布外(OOD)偏移程度发现,随着任务难度增加,LLM的最后隐藏层表示显著稀疏化。通过控制分析和学习动态解释,证明这种稀疏性是稳定推理的适应机制,并设计了基于稀疏性的课程学习策略,显著提升了性能。研究提供了关于LLM如何内部化OOD挑战的新机制性见解。代码库可访问:https://github.com/MingyuJ666/sparsityLLM。
英文摘要:
In this work, we investigate how the internal representations of Large Language Models (LLMs) change with inputs of increasing difficulty. Although sparsity changes may not be obvious for individual samples, dataset-level results reveal a consistent statistical trend: as task difficulty increases (e.g., harder questions, longer contexts, or more answer choices), the last hidden states of LLMs become systematically sparser. In short, \textbf{the harder the task, the sparser the representation}. This sparsity--difficulty relation is observable across diverse models and domains, suggesting that harder inputs drive more concentrated activation patterns in the last hidden state. Through a series of controlled analyses and learning-dynamics experiments, we show that representational density is a learned property of data familiarity: models develop rich, distributed representations for mastered patterns, while unfamiliar inputs default to sparser activations. Our finding is also actionable. We illustrate this with Sparsity-Guided Curriculum In-Context Learning (SG-ICL), which uses sparsity to select few-shot demonstrations matched to the query's difficulty, outperforming standard CoT and Auto-CoT baselines on MATH-500. Our study provides new insights into how LLM representations reflect task difficulty and how this signal can be leveraged for inference. The source code is available at the URL: https://github.com/MingyuJ666/sparsityLLM.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。