具有Wasserstein正则化截断和质量惩罚的几何感知解码
Geometry-Aware Decoding with Wasserstein-Regularized Truncation and Mass Penalties for Large Language Models
AI总结：
本文提出Top-W解码方法，通过Wasserstein距离保持token嵌入空间几何特性，平衡概率质量与熵，提升大语言模型生成质量与创造力。
摘要：
大型语言模型（LLMs）必须在开放生成中平衡多样性与创造性与逻辑一致性。现有基于截断的采样器效果有限，主要依赖概率质量与熵而忽略token空间的语义几何。本文提出Top-W，一种几何感知截断规则，利用Wasserstein距离定义在token嵌入几何上，保持截断分布接近原始分布，同时显式平衡保留的概率质量与保留集合的熵。我们的理论推导出一种简单的闭式结构用于固定势能子集更新：根据质量-熵权衡，最优截断要么坍缩为单个token，要么形成可高效线性扫描找到的一维前缀。我们使用高效的几何基于势能（最近集或k-NN）实现Top-W，并配以交替解码流程，保持标准截断-采样接口不变。在四个基准（GSM8K、GPQA、AlpacaEval和MT-Bench）上对三个指令微调模型的广泛实验表明，Top-W在三个指令微调模型上表现优异，相比现有最先进解码方法提升达33.7%。此外，我们发现Top-W不仅提升了准确性导向的性能，还通过基于评判的开放生成评估提升了创造力。
英文摘要：
Large language models (LLMs) must balance diversity and creativity against logical coherence in open-ended generation. Existing truncation-based samplers are effective but largely heuristic, relying mainly on probability mass and entropy while ignoring semantic geometry of the token space. We present Top-W, a geometry-aware truncation rule that uses Wasserstein distance-defined over token-embedding geometry-to keep the cropped distribution close to the original, while explicitly balancing retained probability mass against the entropy of the kept set. Our theory yields a simple closed-form structure for the fixed-potential subset update: depending on the mass-entropy trade-off, the optimal crop either collapses to a single token or takes the form of a one-dimensional prefix that can be found efficiently with a linear scan. We implement Top-W using efficient geometry-based potentials (nearest-set or k-NN) and pair it with an alternating decoding routine that keeps the standard truncation-and-sampling interface unchanged. Extensive experiments on four benchmarks (GSM8K, GPQA, AlpacaEval, and MT-Bench) across three instruction-tuned models show that Top-W consistently outperforms prior state-of-the-art decoding approaches achieving up to 33.7% improvement. Moreover, we find that Top-W not only improves accuracy-focused performance, but also boosts creativity under judge-based open-ended evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
