arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.35328cs.CLcs.AI
收藏

约束解码:有害还是有益?重新审视其对大语言模型性能的影响

Constrained Decoding: Harmful or Beneficial? Revisiting Its Impact on LLM Performance

Télécom Paris(巴黎电信学院);Institut Polytechnique de Paris(巴黎综合理工学院);Nokia Bell Labs(诺基亚贝尔实验室)

Ngoc Trinh Hung Nguyen, Alonso Silva, Armen Aghasaryan, Mehwish Alam

AI总结:

该研究重新审视约束解码对LLM性能的影响,通过分类实验发现其影响取决于约束类型,有效性与推理约束可提升性能,表观下降源于非等效设置。

摘要:

约束解码可确保大语言模型(LLMs)生成符合预定义格式的输出,但通常被认为会固有地降低推理性能。我们通过三类约束的分类法来检验这一假设:格式约束(强制输出结构)、有效性约束(屏蔽无效答案)、推理约束(限制推理轨迹)。我们从理论上证明,适当的约束可通过消除无效路径来保留或提升性能。在四个模型家族和十一个开源模型上,采用最先进约束解码策略在不同基准上进行的因果消融实验表明,表观性能下降主要源于非等效提示和输出格式,而非解码器级别的屏蔽。同时,有效性约束通过消除无效答案提升可靠性与可提取性,推理约束在强制有益轨迹时可提升推理能力及相应的准确性。这些发现表明,约束解码的影响取决于约束类型及其消除的输出空间。我们的代码可用。

英文摘要:

Constrained decoding guarantees that Large Language Models (LLMs) produce outputs conforming to predefined formats, but is often assumed to inherently degrade reasoning. We examine this assumption through a taxonomy of three constraint types: \emph{format constraints}, which enforce output structure; \emph{validity constraints}, which mask invalid answer; and \emph{reasoning constraints}, which restrict reasoning trajectories. We theoretically show that appropriate constraints can preserve or improve performance by eliminating invalid paths. Across four model families and eleven open-source models, causal ablations on diverse benchmarks using state-of-the-art constrained decoding strategies reveal that apparent performance degradation largely stems from non-equivalent prompting and output formats rather than from decoder-level masking. Meanwhile, validity constraints improve reliability and extractability by eliminating invalid answers, while reasoning constraints can improve reasoning and, consequently, accuracy when they enforce beneficial trajectories. These findings show that the impact of constrained decoding depends on the type of constraint and the output space it removes. Our code is available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑