arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.03708cs.CLcs.PF
收藏

超越标记：基于语义的推测解码用于通过探测内部状态实现高效的推理

Beyond Tokens: Semantic-Aware Speculative Decoding for Efficient Inference by Probing Internal States

Centre for Software Excellence, Huawei, Canada（华为加拿大软件 excellence 中心）；Department of Computer Science, University of Manitoba, Canada（曼尼托巴大学计算机科学系）；School of Computing, Queen’s University, Canada（皇后大学计算学院）

Ximing Dong, Shaowei Wang, Dayi Lin, Boyuan Chen, Ahmed E. Hassan

备注：
EMNLP2026 Main
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

SemanticSpec通过探测模型内部状态，实现基于语义的高效推测解码，提升大型推理模型的推理效率。

摘要：

大型语言模型（LLMs）在许多任务上表现出色，但因自回归解码而面临高推理延迟的问题。这一问题在大型推理模型（LRMs）中尤为严重，因为它们生成长长的思考链。虽然推测解码通过并行生成和验证多个标记来加速推理，但现有方法在标记级别操作，忽略了语义等价性（即不同标记序列表达相同意义），导致效率低下。我们提出了SemanticSpec，一种基于语义的推测解码框架，通过验证整个语义序列而非单个标记来实现更高效的推理。SemanticSpec引入了一种语义概率估计机制，通过探测模型的内部隐藏状态来评估生成具有特定意义的序列的可能性。在四个基准测试中的实验表明，SemanticSpec在DeepSeekR1-32B上实现了高达2.7倍的加速，在QwQ-32B上实现了2.1倍的加速，一致优于基于标记和序列的基线方法在效率和有效性方面。

英文摘要：

Large Language Models (LLMs) achieve strong performance across many tasks but suffer from high inference latency due to autoregressive decoding. The issue is exacerbated in Large Reasoning Models (LRMs), which generate lengthy chains of thought. While speculative decoding accelerates inference by drafting and verifying multiple tokens in parallel, existing methods operate at the token level and ignore semantic equivalence (i.e., different token sequences expressing the same meaning), leading to inefficient rejections. We propose SemanticSpec, a semantic-aware speculative decoding framework that verifies entire semantic sequences instead of tokens. SemanticSpec introduces a semantic probability estimation mechanism that probes the model's internal hidden states to assess the likelihood of generating sequences with specific meanings. Experiments on four benchmarks show that SemanticSpec achieves up to 2.7x speedup on DeepSeekR1-32B and 2.1x on QwQ-32B, consistently outperforming token-level and sequence-level baselines in both efficiency and effectiveness.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑