通过动态拒绝引导增强大语言模型的指令遵循能力
Enhancing Instruction Following of LLMs via Activation Steering with Dynamic Rejection
AI总结：
DIRECTER通过动态调节引导强度和基于可能性的解码循环，提升大语言模型在复杂指令下的准确性和生成质量。
摘要：
大型语言模型（LLMs）尽管在指令调优方面有所进步，但仍常常无法遵循复杂的用户指令。激活引导技术旨在通过操纵模型内部来缓解这一问题，但存在过度引导的潜在风险，即对指令的过度强调会降低任务准确性和整体文本质量。为此，我们引入了DIRECTER（动态拒绝引导），一种新颖的引导方法，通过缩放KV缓存动态调节引导强度，而无需额外数据集。DIRECTER将引导与基于可能性的解码循环相结合，该循环在每一步都通过将引导输出分布与原始分布进行比较，自适应地调整引导强度。如果引导输出被认为不可信，则逐步减弱引导强度。这种强度调节由轻量级的一次性注意力敏感度分析引导，该分析按对模型表示的影响对层进行排序。广泛评估显示，DIRECTER在多样化的基准测试中显著增强了指令遵循能力，比基线提高了高达6.5%的准确性，而无需在生成质量和任务忠实度上产生常见权衡。所提出的动态、基于可能性的控制在激活引导过程中进一步展示了其作为缓解过度引导的一般机制的潜力，该机制与现有基线兼容。
英文摘要：
Large Language Models (LLMs), despite advances in instruction tuning, often fail to follow complex user instructions. Activation steering techniques aim to mitigate this by manipulating model internals, but have a potential risk of oversteering, where excessive emphasis on the instruction degrades task accuracy and overall text quality. To address this, we introduce DIRECTER (Dynamic rejection steering), a novel steering method that dynamically modulates steering strength by scaling the KV cache without extra dataset. DIRECTER couples steering with a plausibility-guided decoding loop, which adaptively adjusts steering strength at each step by comparing the steered output distribution to the original. If the steered output is deemed implausible, steering strength is progressively weakened. This strength modulation is guided by a lightweight, one-time attention sensitivity analysis that ranks layers by their influence on model representations. Extensive evaluations show that DIRECTER significantly enhances instruction-following capabilities across diverse benchmarks, improving accuracy by up to 6.5% over baselines without the common trade-offs in generation quality or task fidelity. The proposed dynamic, plausibility-guided control during activation steering further demonstrates its potential as a general mechanism for mitigating oversteering that is compatible with existing baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
