AI 中文总结
本研究提出无训练的SAKE引导方法，通过核Gram矩阵的二阶Rényi熵动态调整文本扩散模型的采样分布，实现更优的保真度与多样性平衡，提升了代码、数学等推理任务的多样本性能。
AI 中文摘要
尽管扩散模型已通过高质量生成和可控引导机制革新了图像合成等连续领域，但将这种可控性应用于离散、序列性质的文本仍是一个未解决的挑战。同时，当前的采样策略和引导方法仅调整token似然，未捕捉更广泛的语义格局，导致保真度与多样性之间的平衡欠佳。在本研究中，我们提出了一种新颖的无训练语义感知核熵（SAKE）引导方法。该方法在核Gram矩阵上计算二阶Rényi熵，此矩阵既捕捉跨token的语义交互，又捕捉token的相对位置。通过在嵌入空间中线性化该目标，我们推导得出可处理的引导信号，该信号能动态调整采样分布：在冗余时将其展平以鼓励探索，在多样性时将其锐化以提升保真度。实证实验表明，与温度缩放和离散引导基线相比，我们的方法在保真度与多样性之间实现了更优的帕累托前沿，并提升了代码和数学生成等推理密集型任务的多样本性能。
英文摘要
Although diffusion models have revolutionized continuous domains like image synthesis through high quality generations and controllable guidance mechanisms, bringing this controllability to the discrete, sequential nature of text remains an open challenge. Meanwhile, current sampling strategies and guidance methods adjust token likelihoods without capturing the broader semantic landscape, leading to a suboptimal balance between fidelity and diversity. In this work, we introduce a novel training-free Semantic-Aware Kernel Entropy (SAKE) guidance method. Our method computes the order-2 Rényi entropy over a kernel Gram matrix that captures both cross-token semantic interactions and relative token positions. By linearizing this objective in the embedding space, we derive a tractable guidance signal that dynamically adjusts the sampling distribution, flattening it to encourage exploration during redundancy and sharpening it for fidelity when diverse. Empirical experiments demonstrate that our approach achieves a superior Pareto frontier between fidelity and diversity, and improves multi-sample performance on reasoning-intensive tasks, such as code and mathematics generation, compared to temperature scaling and discrete guidance baselines.
