Top-b:自回归生成过程中的熵调控相对概率带
Top-b: Entropic Regulation of Relative Probability Bands in Autoregressive Language Processes
AI总结:
本文提出Top-b方法,通过动态带宽系数调控候选集,减少生成熵和解码方差,同时保持推理准确性,实现自回归生成的自调节控制。
摘要:
概率语言生成器理论上被建模为离散随机过程,但标准解码策略(Top-k,Top-p)施加的静态截断规则无法适应自然语言的动态信息密度。本文将生成过程建模为相对概率流形上的轨迹,引入Top-b(自适应相对带采样)解码策略,通过与模型分布瞬时香农熵严格耦合的动态带宽系数调节候选集。理论框架证明Top-b在尾分布上是方差最小化操作符。在GPQA和GSM8K基准上的实验证实,Top-b显著降低生成熵和解码方差,同时保持竞争性的推理准确性,有效近似自调节控制系统的自回归生成。
英文摘要:
Probabilistic language generators are theoretically modeled as discrete stochastic processes, yet standard decoding strategies (Top-k, Top-p) impose static truncation rules that fail to accommodate the dynamic information density of natural language. This misalignment often forces a suboptimal trade-off: static bounds are either too restrictive for high-entropy creative generation or too permissive for low-entropy logical reasoning. In this work, we formalize the generation process as a trajectory through a relative probability manifold. We introduce Top-b (Adaptive Relative Band Sampling), a decoding strategy that regulates the candidate set via a dynamic bandwidth coefficient coupled strictly to the instantaneous Shannon entropy of the model's distribution. We provide a theoretical framework demonstrating that Top-b acts as a variance-minimizing operator on the tail distribution. Empirical validation on GPQA and GSM8K benchmarks indicates that Top-b significantly reduces generation entropy and inter-decoding variance while maintaining competitive reasoning accuracy, effectively approximating a self-regulating control system for autoregressive generation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。