上下文学习真的学习了吗？重新思考大型语言模型如何通过上下文学习响应和解决任务
Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning
AI总结：
本文将上下文学习（ICL）性能分解为标签空间、格式和判别三维度，发现示范对判别影响甚微但能有效调节标签和格式，且检索语义相似示例可显著提升判别能力，但标签多样性存在权衡。
摘要：
上下文学习（ICL）随着大型语言模型（LLMs）规模扩大而发展，已成为一项强大的能力。通过使用少样本示范示例指导LLMs，ICL使它们无需更新数百万参数即可执行广泛的任务。然而，示范对提升最终任务性能的精确贡献在近期的分析研究中尚未得到彻底调查。本文中，我们在经验上将ICL的整体性能分解为标签空间、格式和判别三个维度，并在多样化的任务上评估了四个通用LLMs。出乎意料的是，我们发现示范在激发语言模型的判别知识方面影响甚微。然而，ICL在调节标签空间和格式方面表现出显著效力，这有助于LLMs响应期望的标签词。我们随后证明，这种能力的作用类似于供LLMs遵循的详细指令。此外，我们还深入分析了检索辅助ICL的机制。我们的发现表明，检索语义相似的示例显著提升了模型的判别能力。然而，我们也观察到在选择良好的上下文示例时，关于标签多样性存在一个权衡。
英文摘要：
In-context Learning (ICL) has emerged as a powerful capability alongside the development of scaled-up large language models (LLMs). By instructing LLMs using few-shot demonstrative examples, ICL enables them to perform a wide range of tasks without updating millions of parameters. However, the precise contributions of demonstrations towards improving end-task performance have not been thoroughly investigated in recent analytical studies. In this paper, we empirically decompose the overall performance of ICL into three dimensions, label space, format, and discrimination, and we evaluate four general-purpose LLMs across a diverse range of tasks. Counter-intuitively, we find that the demonstrations have a marginal impact on provoking discriminative knowledge of language models. However, ICL exhibits significant efficacy in regulating the label space and format, which helps LLMs respond to desired label words. We then demonstrate that this ability functions similar to detailed instructions for LLMs to follow. We additionally provide an in-depth analysis of the mechanism of retrieval helping with ICL. Our findings demonstrate that retrieving the semantically similar examples notably boosts the model's discriminative capability. However, we also observe a trade-off in selecting good in-context examples regarding label diversity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
