arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.01237cs.AI
收藏

为大语言模型的高效推理时间推理进行预测调度

Predictive Scheduling for Efficient Inference-Time Reasoning in Large Language Models

Harvard College(哈佛学院);Harvard SEAS(哈佛工程与应用科学学院)

Katrina Brown, Aneesh Muppidi, Rana Shahout

AI总结:

预测调度通过预运行轻量级预测器优化标记预算分配,提升大语言模型在推理任务中的准确性和效率。

摘要:

大语言模型(LLMs)通过生成多个思维链(CoT)轨迹在复杂的推理任务上实现最先进的准确性,但使用每个查询固定的标记预算会导致在简单输入上过度计算而在困难输入上不足。我们引入了预测调度,这是一种即插即用的框架,通过预运行轻量级预测器,如在中间Transformer隐藏状态上的MLP或在原始问题文本上LoRA微调的分类器,来在任何完整生成之前估计每个查询的最优推理长度或难度。我们的贪婪批量分配器动态地将固定总标记预算分配到查询中,以最大化预期准确性。在GSM8K算术基准测试中,预测调度在相同标记成本下比均匀预算分配获得高达7.9个百分点的绝对准确性提升,接近完美预知的 oracle 的50%差距。系统性的层间研究揭示了Transformer中间层(12-17层)携带最丰富的大小估计信号。这些结果表明,预运行预算预测能够实现计算-准确性权衡的精细控制,为延迟敏感、成本高效的LLM部署提供了具体路径。

英文摘要:

Large language models (LLMs) achieve state-of-the-art accuracy on complex reasoning tasks by generating multiple chain-of-thought (CoT) traces, but using a fixed token budget per query leads to over-computation on easy inputs and under-computation on hard ones. We introduce Predictive Scheduling, a plug-and-play framework that pre-runs lightweight predictors, an MLP on intermediate transformer hidden states or a LoRA-fine-tuned classifier on raw question text, to estimate each query's optimal reasoning length or difficulty before any full generation. Our greedy batch allocator dynamically distributes a fixed total token budget across queries to maximize expected accuracy. On the GSM8K arithmetic benchmark, predictive scheduling yields up to 7.9 percentage points of absolute accuracy gain over uniform budgeting at identical token cost, closing over 50\% of the gap to an oracle with perfect foresight. A systematic layer-wise study reveals that middle layers (12 - 17) of the transformer carry the richest signals for size estimation. These results demonstrate that pre-run budget prediction enables fine-grained control of the compute-accuracy trade-off, offering a concrete path toward latency-sensitive, cost-efficient LLM deployments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑