在指令阶梯上推理以实现可控语言模型
Reasoning Up the Instruction Ladder for Controllable Language Models
AI总结：
通过将指令层级解析重构为推理任务，并构建VerIH数据集进行轻量级强化学习，使模型能优先处理高层级指令，在冲突场景下提升约20%的指令遵循准确率，并增强对越狱和提示注入的鲁棒性。
摘要：
随着基于大型语言模型（LLM）的系统在现实世界决策中承担高风险角色，它们必须在单个提示上下文中协调来自多个来源的竞争指令。强制执行指令层级（其中高层级指令覆盖低优先级请求）对于LLM的可靠性和控制至关重要。在这项工作中，我们将指令层级解析重构为推理任务。模型在生成响应之前必须首先“思考”给定用户提示与更高优先级指令之间的关系。为了实现这一能力，我们构建了VerIH，一个具有可验证答案的约束遵循任务训练数据集，包含一致和冲突的系统-用户指令。我们表明，使用VerIH进行轻量级强化学习有效地将模型的通用推理能力转移到指令优先级排序上。我们的方法在多个模型家族上，在指令遵循和指令层级基准测试中均带来一致的改进，在冲突设置中实现了约20%的绝对提升。我们的方法还改进了对训练分布之外的安全关键场景的对齐，表现出对越狱和提示注入的增强鲁棒性，将绝对攻击成功率降低高达20%。我们的结果确立了指令层级推理作为提高AI可靠性的实用机制，其中对系统提示的有针对性更新会产生可预测、可控且稳健的模型行为变化。
英文摘要：
As large language model (LLM) based systems take on high-stakes roles in real-world decision-making, they must reconcile competing instructions from multiple sources within a single prompt context. Enforcing an instruction hierarchy, where higher-level directives override lower-priority requests, is critical to the reliability and control of LLMs. In this work, we reframe instruction hierarchy resolution as a reasoning task. The model must first "think" about the relationship between a given user prompt and higher-priority instructions before generating a response. To enable this capability, we construct VerIH, a training dataset of constraint-following tasks with verifiable answers, comprising aligned and conflicting system-user instructions. We show that lightweight reinforcement learning with VerIH effectively transfers general reasoning capabilities of models to instruction prioritization. Our method leads to consistent improvements across multiple model families on both instruction following and instruction hierarchy benchmarks, achieving ~20% absolute improvement in conflict setups. Our method also leads to improved alignment to safety-critical scenarios beyond the training distribution, exhibiting increased robustness against jailbreak and prompt injection, reducing absolute attack success rates by up to 20%. Our results establish reasoning over instruction hierarchies as a practical mechanism for improving AI reliability, where targeted updates to system prompts produce predictable, controllable, and robust changes in model behavior.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
