上下文学习的奥秘：关于解释与分析的全面综述
The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysis
AI总结：
本文从理论和实证两个角度系统综述了上下文学习（ICL）的解释与分析进展，梳理其数学基础、机制可解释性及影响因素，并指出挑战与未来方向。
摘要：
理解使大型语言模型（LLMs）能够通过演示示例获得卓越能力的上下文学习（ICL）能力至关重要。这种重要性不仅源于在各种任务中更好地利用这一能力，还源于主动识别和缓解可能随之而来的潜在风险，包括对真实性、偏见和毒性等方面的担忧。在本文中，我们对上下文学习的解释与分析进行了全面的综述。首先，我们简要介绍了上下文学习的背景和定义。然后，我们从两个角度概述了相关进展：1）理论角度，重点研究机制可解释性，并深入探讨ICL背后的数学基础；2）实证角度，涉及对与ICL相关因素进行实证分析的研究。最后，我们强调了所面临的挑战，并提出了未来研究的潜在方向。我们相信，我们的工作为进一步探索上下文学习的解释奠定了基础。此外，我们还创建了一个包含本综述所引用资源的仓库。
英文摘要：
Understanding in-context learning (ICL) capability that enables large language models (LLMs) to excel in proficiency through demonstration examples is of utmost importance. This importance stems not only from the better utilization of this capability across various tasks, but also from the proactive identification and mitigation of potential risks, including concerns regarding truthfulness, bias, and toxicity, that may arise alongside the capability. In this paper, we present a thorough survey on the interpretation and analysis of in-context learning. First, we provide a concise introduction to the background and definition of in-context learning. Then, we give an overview of advancements from two perspectives: 1) a theoretical perspective, emphasizing studies on mechanistic interpretability and delving into the mathematical foundations behind ICL; and 2) an empirical perspective, concerning studies that empirically analyze factors associated with ICL. We conclude by highlighting the challenges encountered and suggesting potential avenues for future research. We believe that our work establishes the basis for further exploration into the interpretation of in-context learning. Additionally, we have created a repository containing the resources referenced in our survey.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
