预训练大语言模型的固有局限：指令微调与上下文学习能力的意外趋同
The Inherent Limits of Pretrained LLMs: The Unexpected Convergence of Instruction Tuning and In-Context Learning Capabilities
AI总结：
该研究通过对90个大语言模型开展跨模型家族、规模与任务类型的实验，发现指令微调模型性能与基础模型上下文学习性能显著相关，指出预训练数据是二者能力的共同限制边界，拓展了对大语言模型能力构成的认知。
摘要：
在海量网页级语料上训练的大语言模型（LLM）已在各类任务中展现出卓越能力，模型规模扩大时表现尤为突出。然而，即便最先进的模型在某些场景下仍会表现不佳，有时甚至无法解决幼儿都能处理的问题，这表明传统的任务复杂度概念不足以解释LLM的能力边界。但探索LLM能力的工作因一个事实变得复杂：大多数广泛使用的模型都经过“指令微调”，以对提示做出恰当响应。为厘清影响LLM性能的各类因素，我们研究了经过指令微调的模型，与使用上下文示例进行提示的基础模型相比，是否具备本质上不同的能力。我们在不同模型家族、模型规模和任务类型上开展了大量实验，其中包括对90个不同LLM进行指令微调，结果表明指令微调模型的性能与其对应基础模型的上下文学习性能显著相关。通过明确指令微调的实际作用，我们拓展了此前关于上下文学习的研究——该研究指出基础模型会利用预训练数据中的先验知识解决任务。具体而言，我们将这一认知拓展至指令微调模型，认为其预训练数据同样为可解决的任务设定了限制边界，指令微调数据集则会产生额外影响。
英文摘要：
Large Language Models (LLMs), trained on extensive web-scale corpora, have demonstrated remarkable abilities across diverse tasks, especially as they are scaled up. Nevertheless, even state-of-the-art models struggle in certain cases, sometimes failing at problems solvable by young children, indicating that traditional notions of task complexity are insufficient for explaining LLM capabilities. However, exploring LLM capabilities is complicated by the fact that most widely-used models are also "instruction-tuned" to respond appropriately to prompts. With the goal of disentangling the factors influencing LLM performance, we investigate whether instruction-tuned models possess fundamentally different capabilities from base models that are prompted using in-context examples. Through extensive experiments across various model families, scales and task types, which included instruction tuning 90 different LLMs, we demonstrate that the performance of instruction-tuned models is significantly correlated with the in-context performance of their base counterparts. By clarifying what instruction-tuning contributes, we extend prior research into in-context learning, which suggests that base models use priors from pretraining data to solve tasks. Specifically, we extend this understanding to instruction-tuned models, suggesting that their pretraining data similarly sets a limiting boundary on the tasks they can solve, with the added influence of the instruction-tuning dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
