Entropy-UID:一种优化信息密度的方法
Entropy-UID: A Method for Optimizing Information Density
AI总结:
本文提出 Entropy-UID token 选择方法,通过联合最小化熵与惊异值落实 UID 原则,在多个基准上相较 GPT-2 和启发式方法降低信息波动并改善生成平衡性。
摘要:
平衡且高效的信息流对于优化语言生成模型至关重要。在本工作中,我们提出 Entropy-UID,这是一种新的 token 选择方法,通过平衡熵与统一信息密度(Uniform Information Density,UID)原则来提升文本生成效率。我们的方法通过联合最小化熵和惊异值(surprisal)来自适应调整 token 选择,从而促进生成序列中更均匀的信息分布。理论验证表明,Entropy-UID 能在保持流畅性和连贯性的同时,最优地减少信息尖峰。该方法已在 WikiText-2、OpenWebText 和 WMT 等多个基准数据集上使用信息论指标进行评估。实验结果表明,与标准 GPT-2 及其他启发式方法相比,Entropy-UID 实现了更低的惊异值和熵方差,从而带来更平衡、更接近人类的文本生成。我们的发现表明,利用信息论约束来改进自回归语言模型中的 token 选择策略具有潜力。
英文摘要:
Balanced and efficient information flow is essential for optimizing language generation models. In this work, we propose Entropy-UID, a new token selection method that balances entropy and Uniform Information Density (UID) principles for enhanced efficiency of text generation. Our approach adaptively adjusts token selection by jointly minimizing entropy and surprisal, promoting more even information distribution across generated sequences. Theoretical validation demonstrates that Entropy-UID optimally reduces information spikes while maintaining fluency and coherence. The method has been evulated using information-theoretic metrics on multiple benchmark datasets, including WikiText-2, OpenWebText, and WMT. Experimental results show that Entropy-UID achieves lower surprisal and entropy variance compared to standard GPT-2 and alternative heuristics, leading to more balanced and human-like text generation. Our findings point towards the potential of leveraging information-theoretic constraints to refine token selection strategies in autoregressive language models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。