arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.22672cs.AI
收藏

能力是负担吗?更强大的语言模型在关键时刻做出更差的预测

Is Capability a Liability? More Capable Language Models Make Worse Forecasts When It Matters Most

Forecasting Research Institute(预测研究 institute)

Nick Merrill, Jaeho Lee, Ezra Karger

AI总结:

本文发现,在时间序列呈现超线性增长和制度转换尾部风险的预测问题上,更强大的语言模型反而产生更差的分位数预测,并分析了这一逆缩放现象的机制、影响因素及评估指标问题。

摘要:

我们记录了LLM在预测问题上的逆缩放现象,这些问题的底层时间序列表现出超线性增长和制度转换的尾部风险,这种结构在金融和流行病学中很常见。在这些任务上,更强大的模型会产生更差的分位数预测。该模式出现在我们发布的、无污染的模拟世界基准ForecastBench-Sim(FBSim)上,在预测具有匹配线性控制的合成SIR流行病时,并在COVID-19、麻疹、住房市场和恶性通货膨胀的真实世界数据集中得到复现。每个分位数的分解表明,失败集中在尾部上端,更强大的模型将其向上移动以跟踪激进的增长外推,而下尾部保持不变。Llama-3.1的族内研究表明,模型规模和后训练都独立地促成了这种效应。领域知识并不能可靠地挽救校准。这种逆缩放并不出现在LLM预测基准中常见的单阈值指标上,在相同的输出上,能力-准确性关系的符号发生了反转。在常规截止点上的单阈值评分忽略了尾部上端的成本;包含尾部的评分在相同的输出上反转了能力-准确性关系的符号。我们建议LLM预测评估使用连续(且无界)的准确性度量以及有界的二元阈值度量。

英文摘要:

We document inverse scaling in LLMs on forecasting problems whose underlying time series exhibit superlinear growth and tail risk of regime change, a structure common in finance and epidemiology. On these tasks, more capable models produce worse distributional forecasts. The pattern appears on ForecastBench-Sim (FBSim), a contamination-free, simulated-world benchmark we release, in forecasting synthetic SIR epidemics with a matched linear control, and replicates in real-world datasets on COVID-19, measles, housing markets, and hyperinflation. A per-quantile decomposition shows the failure concentrates at the upper tail, which more capable models shift upward to track aggressive extrapolations of growth, while the lower tail stays put. A within-family study of Llama-3.1 shows that both model scale and post-training independently contribute to this effect. Domain knowledge does not reliably rescue calibration. This inverse scaling does not appear on single-threshold metrics common in LLM forecasting benchmarks, reversing the sign of the capability--accuracy relationship on identical outputs. Single-threshold scoring at conventional cutoffs misses the upper-tail cost; tail-inclusive scoring reverses the sign of the capability--accuracy relationship on the same outputs. We recommend that LLM forecasting evaluations use continuous (and unbounded) measures of accuracy alongside bounded binary threshold metrics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑