Power-SMC:低延迟序列级功率采样用于训练自由LLM推理
Power-SMC: Low-Latency Sequence-Level Power Sampling for Training-Free LLM Reasoning
AI总结:
本文提出Power-SMC,一种低延迟的序列级功率采样方法,用于训练自由的大语言模型推理,通过并行粒子集和重要权重修正提升推理效率。
摘要:
许多近期大语言模型推理能力的提升可以解释为分布锐化:将生成偏向于由预训练模型支持的高似然轨迹,而不是修改其权重。自然形式化是序列级功率分布π_α(y|x)∝p_θ(y|x)^α(α>1),该分布将质量集中在完整序列而非调整token级温度。先前工作表明,从该分布中使用Metropolis-Hastings(MH)采样可恢复强大的推理性能,但导致数量级的推理延迟增加。本文引入Power-SMC,一种无训练的顺序蒙特卡洛方案,旨在达到相同的目标同时保持接近标准解码延迟。Power-SMC并行推进小粒子集,逐token修正重要权重,并在必要时重新采样,所有操作均在单个GPU友好的批量解码中完成。我们证明温度τ=1/α是唯一前缀仅提案,最小化增量权重方差,通过前缀条件化Rényi熵解释残差不稳定性,并引入指数桥接计划,提高粒子稳定性而不改变目标。在MATH500上,Power-SMC匹配或超过MH功率采样,同时将延迟从16-28倍降低到1.4-3.3倍。代码可在https://github.com/ArminAzizi98/Power-SMC获取。
英文摘要:
Reasoning ability in large language models is often attributed to \emph{distribution sharpening}: concentrating output probability on high-likelihood sequences. Recent works show that this sharpening effect can be obtained at inference time, without modifying model parameters, and can elicit strong reasoning performance. A natural formalization is the \emph{sequence-level power distribution}, which is proportional to the model's probability raised to an exponent $α>1$. Prior work leveraged Metropolis--Hastings (MH) sampling to draw samples from this distribution and achieves strong results, however, at order-of-magnitude inference slowdowns. We introduce \textbf{Power-SMC}, a \textit{`training-free'} sampling method that targets the same power distribution yielding close to standard decoding latency. Power-SMC maintains multiple candidate sequences in parallel. Each candidate sequence is assigned a score, namely the \emph{importance weight}, that measures how well it matches the power distribution. It then periodically prunes low-scoring candidate sequences in favor of high-scoring ones. We further provide a theoretical justification for the design choices in Power-SMC. Among all next-token sampling strategies that do not rely on future tokens, we prove that sampling temperature $τ{=}1/α$ uniquely eliminates per-step weight variance. Finally, we characterize the remaining source of weight instability and introduce a gradual sharpening schedule to reduce weight collapse, while targeting the same power distribution. Extensive evaluations on MATH500, GSM8K, GPQA, and HumanEval show that Power-SMC matches or exceeds MH sampling in accuracy, preserves output diversity unlike RL-finetuned models, while \textbf{accelerating inference speed by up to} {$\mathbf{17.6}\times$}. The code is available at https://github.com/ArminAzizi98/Power-SMC.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。