顶部H解码：在受限制的熵下适应创造力和连贯性
Top-H Decoding: Adapting the Creativity and Coherence with Bounded Entropy in Text Generation
AI总结：
本文提出top-H解码，通过熵约束最小散度问题提升文本生成的创造力与连贯性，实验证明其在创意写作任务中优于现有方法，且在问答数据集上表现稳健。
摘要：
大型语言模型（LLMs）尽管在广泛任务中表现出色，但在开放性文本生成中常难以平衡多样性与创造力和逻辑连贯性。现有截断采样技术如温度缩放、top-p（核采样）和min-p采样旨在管理这一权衡，但存在局限，特别是在有效整合模型置信度到采样策略方面。为此，本文提出top-H解码。我们首先建立了截断采样中创造力与连贯性相互作用的理论基础，通过建立熵约束最小散度问题。然后证明该最小化问题等价于熵约束质量最大化（ECMM）问题，后者是NP难问题。最后，我们提出top-H解码，一种计算高效的贪心算法来解决ECMM问题。广泛的经验评估显示，top-H在创意写作基准上比现有最先进方法min-p采样高出25.63%，同时在问题回答数据集如GPQA、GSM8K和MT-Bench上保持稳健。此外，LLM-as-judge评估确认top-H即使在较高温度下也能生成连贯的输出，其中创造力尤其关键。总之，top-H在开放性文本生成中推进了最先进水平，可轻松集成到创意写作应用中。代码可在https://github.com/ErfanBaghaei/Top-H-Decoding获取。
英文摘要：
Large language models (LLMs), despite their impressive performance across a wide range of tasks, often struggle to balance two competing objectives in open-ended text generation: fostering diversity and creativity while preserving logical coherence. Existing truncated sampling techniques, including temperature scaling, top-\$p\$ (nucleus) sampling, and min-\$p\$ sampling, aim to manage this trade-off. However, they exhibit limitations, particularly in the effective incorporation of the confidence of the model into the corresponding sampling strategy. For example, min-\$p\$ sampling relies on a single top token as a heuristic for confidence, eventually underutilizing the information of the probability distribution. Toward effective incorporation of the confidence of the model, in this paper, we present **top-H** decoding. We first establish the theoretical foundation of the interplay between creativity and coherence in truncated sampling by formulating an **entropy-constrained minimum divergence** problem. We then prove this minimization problem to be equivalent to an **entropy-constrained mass maximization** (ECMM) problem, which is NP-hard. Finally, we present top-H decoding, a computationally efficient greedy algorithm to solve the ECMM problem. Extensive empirical evaluations demonstrate that top-H outperforms the state-of-the-art (SoTA) alternative of min-\$p\$ sampling by up to **25.63%** on creative writing benchmarks, while maintaining robustness on question-answering datasets such as GPQA, GSM8K, and MT-Bench. Additionally, an *LLM-as-judge* evaluation confirms that top-H indeed produces coherent outputs even at higher temperatures, where creativity is especially critical. In summary, top-H advances SoTA in open-ended text generation and can be *easily integrated* into creative writing applications. The code is available at https://github.com/ErfanBaghaei/Top-H-Decoding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
