上下文向量：通过潜空间引导使上下文学习更有效且更可控
In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering
AI总结：
提出上下文向量（ICV）方法，通过潜空间引导替代传统提示演示，将上下文学习转化为可调幅度的向量，在提升效果与可控性的同时降低计算成本，并支持多指令灵活组合。
摘要：
大型语言模型（LLMs）展现出涌现的上下文学习能力，能够基于示例演示适应新任务。然而，上下文学习在许多场景中效果有限，难以定量控制，且占用上下文窗口空间。为克服这些局限，我们提出一种替代方法，将上下文学习重新转化为上下文向量（ICV）。使用ICV分为两步：首先对演示示例进行一次前向传播，从LLM的潜在嵌入中创建上下文向量。该向量捕获了目标任务的关键信息。对于新查询，我们不再将演示添加到提示中，而是利用ICV调整LLM的潜在状态。ICV方法具有以下优势：1）使LLM更有效地遵循演示示例；2）通过调整ICV的幅度即可轻松控制；3）通过移除上下文演示缩短了提示长度；4）在计算上比微调高效得多。我们证明，在包括安全性、风格转换、角色扮演和格式化在内的多种任务中，与标准上下文学习和微调相比，ICV取得了更好的性能。此外，我们展示了通过对相应的ICV进行简单的向量算术运算，可以灵活地教导LLM同时遵循不同类型的指令。
英文摘要：
Large language models (LLMs) demonstrate emergent in-context learning capabilities, where they adapt to new tasks based on example demonstrations. However, in-context learning has seen limited effectiveness in many settings, is difficult to quantitatively control and takes up context window space. To overcome these limitations, we propose an alternative approach that recasts in-context learning as in-context vectors (ICV). Using ICV has two steps. We first use a forward pass on demonstration examples to create the in-context vector from the latent embedding of the LLM. This vector captures essential information about the intended task. On a new query, instead of adding demonstrations to the prompt, we shift the latent states of the LLM using the ICV. The ICV approach has several benefits: 1) it enables the LLM to more effectively follow the demonstration examples; 2) it's easy to control by adjusting the magnitude of the ICV; 3) it reduces the length of the prompt by removing the in-context demonstrations; 4) ICV is computationally much more efficient than fine-tuning. We demonstrate that ICV achieves better performance compared to standard in-context learning and fine-tuning on diverse tasks including safety, style transfer, role-playing and formatting. Moreover, we show that we can flexibly teach LLM to simultaneously follow different types of instructions by simple vector arithmetics on the corresponding ICVs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
