arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.02212cs.CL
收藏

关于理解上下文学习近期进展的综述

A Survey to Recent Progress Towards Understanding In-Context Learning

Michigan State University(密歇根州立大学);Rensselaer Polytechnic Institute(伦斯勒理工学院)

Haitao Mao, Guangliang Liu, Yao Ma, Rongrong Wang, Kristen Johnson, Jiliang Tang

AI总结:

本文从数据生成视角系统综述上下文学习机制,区分技能识别与技能学习,分析其优劣与共性,为未来研究指明方向。

摘要:

上下文学习(ICL)赋予大型语言模型(LLMs)从提示中提供的少量示例中学习的能力,无需梯度更新即可实现下游泛化。尽管取得了令人鼓舞的经验性成功,ICL的底层机制仍不清晰。现有研究因各种观点而存在模糊性,利用直觉驱动和临时的技术解决方案来解释ICL。在本文中,我们利用数据生成的视角从系统角度重新解释近期的努力,展示这些流行技术解决方案的潜在更广泛用途。在概念定义上,我们严格采用技能识别和技能学习的术语。技能识别选择在预训练期间先前见过的一个已学习数据生成函数,而技能学习可以从上下文数据中学习新的数据生成函数。此外,我们提供了对两种能力优势和劣势的见解,通过数据生成的视角强调它们的共性。该分析为未来研究提出了潜在方向。

英文摘要:

In-Context Learning (ICL) empowers Large Language Models (LLMs) with the ability to learn from a few examples provided in the prompt, enabling downstream generalization without the requirement for gradient updates. Despite encouragingly empirical success, the underlying mechanism of ICL remains unclear. Existing research remains ambiguous with various viewpoints, utilizing intuition-driven and ad-hoc technical solutions to interpret ICL. In this paper, we leverage a data generation perspective to reinterpret recent efforts from a systematic angle, demonstrating the potential broader usage of these popular technical solutions. For a conceptual definition, we rigorously adopt the terms of skill recognition and skill learning. Skill recognition selects one learned data generation function previously seen during pre-training while skill learning can learn new data generation functions from in-context data. Furthermore, we provide insights into the strengths and weaknesses of both abilities, emphasizing their commonalities through the perspective of data generation. This analysis suggests potential directions for future research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑