arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.21215cs.CLcs.AI
收藏

推理时对齐 via 稀疏节点引导

Inference-time Alignment via Sparse Junction Steering

Nanyang Technological University(南洋理工大学);Zhejiang University(浙江大学);G Labs(0G实验室)

Runyi Hu, Jie Zhang, Shiqian Zhao, Jiale Meng, Jiwei Li, Jason Zeng, Ming Wu, Michael Heinrich, Yonggang Wen, Tianwei Zhang

AI总结:

通过稀疏节点引导方法,实现更高效的推理过程对齐,减少计算开销并提升生成质量。

摘要:

在推理过程中对齐通过稀疏节点引导的方法已经 emerged 为一种关键的手段,使得能够通过调节大型语言模型的输出分布来实现细粒度的控制,而无需参数更新。虽然有效,但现有方法依赖于在每一个解码步骤中进行密集干预。这种持续的操控不仅带来了显著的计算开销,还可能通过过度偏离模型的内在分布而损害生成质量。在本工作中,我们证明密集干预是不必要的,并提出稀疏推理时间对齐(SIA),通过仅在生成轨迹中的关键决策点进行干预来实现稀疏节点引导。我们的关键见解是高熵节点标记了生成轨迹中的关键决策点,并且特别容易受到对齐问题的影响,表明需要在这些点引入与对齐相关的奖励信号。在不同模型家族和对齐目标上的广泛实验表明,仅引导20%至80%的token能够实现更优的对齐效率权衡。对于强大的基础模型如Qwen3,干预仅20%的token可以匹配或甚至超越大量后训练的指导模型。这种稀疏性使指导更强,同时更好地保留模型的原生分布,无缝集成到基于搜索的方法如Best-of-N中,并将计算成本降低高达6倍。

英文摘要:

Token-level steering has emerged as a pivotal approach for inference-time alignment, enabling fine-grained control over large language models (LLMs) by modulating their output distributions without parameter updates. While effective, existing methods rely on dense intervention at every decoding step. This persistent manipulation not only incurs substantial computational overhead but also risks compromising generation quality by excessively drifting from the model's intrinsic distribution. In this work, we show that dense intervention is unnecessary and propose Sparse Inference-time Alignment (SIA), which performs sparse junction steering, intervening only at critical decision points along the generation trajectory. Our key insight is that high-entropy junctions are empirically consistent with pivotal decision points in the generation trajectory and are particularly susceptible to misalignment, suggesting that these points benefit from alignment-related reward signal. Extensive experiments across different model families and alignment objectives show that steering only 20%-80% of tokens achieves the superior alignment-efficiency trade-offs. For strong base models such as Qwen3, intervening on as few as 20% of tokens matches or even surpasses heavily post-trained instruct models. This sparsity enables stronger guidance while better preserving the model's native distribution, integrates seamlessly with search-based methods (e.g., Best-of-N), and improves efficiency by reducing the required sampling budget while yielding practical wall-clock speedups under matched-quality comparisons.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑