下一词感知分数：对您的LLM感知能力的分析性评估
Next Token Perception Score: Analytical Assessment of your LLM Perception Skills
AI总结：
本研究提出下一词感知分数（NTPS），用于量化自回归预训练与下游感知任务间的对齐程度，并证明其能预测LoRA微调带来的性能提升，为评估LLM感知能力提供理论依据和实用工具。
摘要：
自回归预训练已成为在大语言模型（LLMs）中学习通用表示的主流范式。然而，线性探针在下游感知任务上的表现显示出显著变异性，这表明为下一词预测优化的特征并不能一致地迁移到下游感知任务中。我们证明，通过自回归学习到的表示所捕获的特征可能位于对感知最具信息量的子空间之外。为了量化自回归预训练与下游感知之间的（不）对齐程度，我们引入了下一词感知分数（NTPS）——一个在线性设置下推导的分数，用于衡量自回归与感知特征子空间之间的重叠。该指标可以轻松地从预训练表示和标注数据中以闭式形式计算，并被证明能够对额外损失提供上界和下界。实验上，我们展示了NTPS在12个多样化的NLP数据集和8个参数范围从270M到8B的预训练模型上与线性探针准确率高度相关，证实了其作为对齐度量的实用性。此外，我们表明NTPS在低秩适应（LoRA）微调后增加，尤其是在大型模型中，这表明LoRA将表示对齐到感知任务增强了子空间重叠，从而提高了下游性能。更重要的是，我们发现NTPS能够可靠地预测LoRA微调所获得的额外准确率提升，从而为LoRA适应提供了一个轻量级的预筛选工具。我们的结果提供了理论见解和实用工具，用于分析性评估LLM的感知能力。
英文摘要：
Autoregressive pretraining has become the de facto paradigm for learning general-purpose representations in large language models (LLMs). However, linear probe performance across downstream perception tasks shows substantial variability, suggesting that features optimized for next-token prediction do not consistently transfer well to downstream perception tasks. We demonstrate that representations learned via autoregression capture features that may lie outside the subspaces most informative for perception. To quantify the (mis)alignment between autoregressive pretraining and downstream perception, we introduce the Next Token Perception Score (NTPS)-a score derived under a linear setting that measures the overlap between autoregressive and perception feature subspaces. This metric can be easily computed in closed form from pretrained representations and labeled data, and is proven to both upper- and lower-bound the excess loss. Empirically, we show that NTPS correlates strongly with linear probe accuracy across 12 diverse NLP datasets and eight pretrained models ranging from 270M to 8B parameters, confirming its utility as a measure of alignment. Furthermore, we show that NTPS increases following low-rank adaptation (LoRA) fine-tuning, especially in large models, suggesting that LoRA aligning representations to perception tasks enhances subspace overlap and thus improves downstream performance. More importantly, we find that NTPS reliably predicts the additional accuracy gains attained by LoRA finetuning thereby providing a lightweight prescreening tool for LoRA adaptation. Our results offer both theoretical insights and practical tools for analytically assessing LLM perception skills.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
