Forecast-Dojo：用于基准测试和训练LLM预测智能体的可重放环境

中文摘要 AI 辅助

我们推出了Forecast-Dojo，这是一个用于基准测试和训练LLM预测智能体的可重放环境。它将已解决的市场预测问题与带日期的新闻相结合，使智能体能够研究事件并在连续的历史日期重新审视其预测。相同的任务和工具支持重复评估、训练交互的收集以及从记录结果中获得的反馈，而无需等待新事件解决。Forecast-Dojo包含1,568个Polymarket事件，按时间划分为训练期和评估期，以及1,880万篇带日期的新闻文章。在对12个模型的评估中，研究工具降低了所有12个模型的Brier分数。随着事件的展开，预测也会改善，在记录更多新日期证据的步骤中收益最大。每个模型在Brier分数和准确性方面仍然落后于历史市场预测。在日期之间携带的信念笔记本降低了研究成本，但并未持续提高预测质量。除了评估之外，Forecast-Dojo还为智能体学习提供了交互轨迹和结果反馈，并以监督微调作为概念验证。

英文摘要

We introduce Forecast-Dojo, a replayable environment for benchmarking and training LLM forecasting agents. It combines resolved prediction-market questions with dated news, allowing agents to research an event and revisit their predictions at successive historical dates. The same tasks and tools support repeated evaluation, collection of training interactions, and feedback from recorded outcomes without waiting for new events to resolve. Forecast-Dojo contains 1,568 Polymarket events, split by time into training and evaluation periods, and 18.8M dated news articles. In an evaluation of 12 models, research tools lower Brier score for all 12. Forecasts also improve as events unfold, with the largest gains at steps where more newly dated evidence is recorded. Every model still trails historical market forecasts in both Brier score and accuracy. A belief notebook carried between dates lowers research cost but does not consistently improve forecast quality. Beyond evaluation, Forecast-Dojo provides interaction trajectories and outcome feedback for agent learning, with supervised fine-tuning as a proof of concept.

发表机构

• Georgia Institute of Technology（佐治亚理工学院）
• Amazon（亚马逊）
• University of Florida（佛罗里达大学）

机构由 AI 辅助整理，请以论文原文为准。