arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.07931cs.LG
收藏

鲁棒长度预测:从重尾提示条件分布的角度

Robust Length Prediction: A Perspective from Heavy-Tailed Prompt-Conditioned Distributions

State Key Laboratory of Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院)

Jing Wang, Yu-Yang Qian, Ke Xue, Chao Qian, Peng Zhao, Zhi-Hua Zhou

AI总结:

本文提出通过重尾提示条件分布进行鲁棒长度预测,改进传统单次采样方法,提升预测稳定性与准确性。

摘要:

输出长度预测对于高效的LLM服务至关重要,因为它直接影响批处理、内存预留和调度。对于仅提示的长度预测,大多数现有方法使用一次采样的长度作为标签,隐式地将每个提示视为有一个真正的目标长度。我们证明这不可靠:即使在固定的模型和解码设置下,相同的提示会诱导一个提示条件的输出长度分布,而不是确定性标量,并且这种分布与重尾行为一致。受此启发,我们将长度预测视为从重尾提示条件长度分布中进行稳健估计。我们提出提示条件长度分布(ProD)方法,这些方法从同一提示的多个独立生成中构建训练目标。两种变体被开发出来以重用服务LLM的隐藏状态:ProD-M,它使用基于中位数的目标进行稳健点预测,以及ProD-D,它使用分布目标以保留提示条件的不确定性。我们通过分析代理模型下的估计误差提供了理论依据。在多样化的场景实验中展示了预测质量的一致提升。

英文摘要:

Output-length prediction is important for efficient LLM serving, as it directly affects batching, memory reservation, and scheduling. For prompt-only length prediction, most existing methods use a one-shot sampled length as the label, implicitly treating each prompt as if it had one true target length. We show that this is unreliable: even under a fixed model and decoding setup, the same prompt induces a \emph{prompt-conditioned output length distribution}, not a deterministic scalar, and this distribution is consistent with \emph{heavy-tailed} behavior. Motivated by this, we cast length prediction as robust estimation from heavy-tailed prompt-conditioned length distributions. We propose prompt-conditioned length distribution (ProD) methods, which construct training targets from multiple independent generations of the same prompt. Two variants are developed to reuse the served LLM's hidden states: \mbox{ProD-M}, which uses a median-based target for robust point prediction, and ProD-D, which uses a distributional target that preserves prompt-conditioned uncertainty. We provide theoretical justifications by analyzing the estimation error under a surrogate model. Experiments across diverse scenarios show consistent gains in prediction quality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑