通过引导标记对大型语言模型进行组合引导
Compositional Steering of Large Language Models with Steering Tokens
AI总结：
本文提出组合引导标记，用于多行为引导，通过自蒸馏将自然语言指令嵌入专用标记，实现更有效的零样本组合，实验表明其在可验证约束上的表现优于其他方法。
摘要：
在现实世界中部署LLMs需要满足多重需求的可控输出。尽管现有工作广泛研究了单行为引导，但组合引导（即同时引导LLMs向多个行为）仍是一个未被探索的问题。本文提出组合引导标记用于多行为引导。我们首先通过自蒸馏将个体行为（以自然语言指令表达）嵌入专用标记。与大多数先前工作在激活空间中操作不同，我们的行为引导生活在输入标记空间中，从而更有效地进行零样本组合。然后我们训练一个专用的组合标记，以配对行为进行训练，并展示其成功捕捉组合概念：它能很好地泛化到未见过的组合，包括包含未见过行为以及未见过的组合数量的组合。我们在不同LLM架构上的实验表明，引导标记在可验证约束（如长度、格式、结构、语言）的多行为引导上优于竞争方法（指令、激活引导和LoRA合并）。此外，我们还展示了引导标记与自然语言指令的互补性，其组合进一步带来了收益。
英文摘要：
Deploying LLMs in real-world applications requires controllable output that satisfies multiple desiderata at the same time. While existing work extensively addresses LLM steering for a single behavior, \textit{compositional steering} -- i.e., steering LLMs simultaneously towards multiple behaviors -- remains an underexplored problem. In this work, we propose \emph{compositional steering tokens} for multi-behavior steering. We first embed individual behaviors, expressed as natural language instructions, into dedicated tokens via self-distillation. Contrary to most prior work, which operates in the activation space, our behavior steers live in the space of input tokens, enabling more effective zero-shot composition. We then train a dedicated \textit{composition token} on pairs of behaviors and show that it successfully captures the notion of composition: it generalizes well to \textit{unseen} compositions, including those with unseen behaviors as well as those with an unseen \textit{number} of behaviors. Our experiments across different LLM architectures show that steering tokens lead to superior multi-behavior steering of verifiable constraints (e.g., length, format, structure, language) compared to competing approaches (instructions, activation steering, and LoRA merging). Moreover, we show that steering tokens complement natural language instructions, with their combination resulting in further gains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
