arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.16007cs.CL
收藏

标签词作为上下文学习中的局部任务向量

Label Words as Local Task Vectors in In-Context Learning

Institute of Neuroscience, Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Shanghai, China(中国科学院神经科学研究所、脑认知与脑启发智能技术重点实验室、脑科学与智能技术卓越中心、中国科学院、上海,中国)

Bowen Zheng, Ming Ma, Zhongqiao Lin, Tianming Yang

AI总结:

本文研究了上下文学习中局部任务向量的作用,发现其在某些任务中能形成全局任务向量,从而提升模型性能。

摘要:

大型语言模型(LLMs)已展现出显著的能力,其中最重要的之一是在上下文学习(ICL)中。通过ICL,LLMs可以从少量示例中推导出潜在规则,并提供符合该规则的答案。先前的研究假设在ICL过程中,网络会在特定位置创建任务向量。该任务向量可以通过对数据集进行平均计算得到。它传达了整体任务信息,因此可以被视为全局的。通过修补全局任务向量,LLMs能够使用虚拟输入实现零样本性能,其效果与少量样本学习相当。然而,我们发现这种全局任务向量并不存在于所有任务中,尤其是在依赖于从多个示例中推断出的规则的任务中,例如分类任务。相反,每个示例提供的信息首先传输到其答案位置,并形成与示例相关的局部任务向量。在某些任务中但不在分类任务中,所有示例的局部任务向量在后期层中汇聚,形成全局任务向量。我们进一步表明,局部任务向量编码了从示例中提取的规则的高层次抽象。我们的研究为LLMs中的ICL机制提供了新的见解,展示了通过信息聚合机制如何实现ICL。

英文摘要:

Large Language Models (LLMs) have demonstrated remarkable abilities, one of the most important being in-context learning (ICL). With ICL, LLMs can derive the underlying rule from a few demonstrations and provide answers that comply with the rule. Previous work hypothesized that the network creates a task vector in specific positions during ICL. The task vector can be computed by averaging across the dataset. It conveys the overall task information and can thus be considered global. Patching the global task vector allows LLMs to achieve zero-shot performance with dummy inputs comparable to few-shot learning. However, we find that such a global task vector does not exist in all tasks, especially in tasks that rely on rules that can only be inferred from multiple demonstrations, such as categorization tasks. Instead, the information provided by each demonstration is first transmitted to its answer position and forms a local task vector associated with the demonstration. In some tasks but not in categorization tasks, all demonstrations' local task vectors converge in later layers, forming the global task vector. We further show that local task vectors encode a high-level abstraction of rules extracted from the demonstrations. Our study provides novel insights into the mechanism underlying ICL in LLMs, demonstrating how ICL may be achieved through an information aggregation mechanism.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑