Forecasting is a task that is difficult to evaluate: the ground truth can only be known in the future. Recent work showing LLM forecasters rapidly approaching human-level performance begs the question: how can we benchmark and evaluate these forecasters instantaneously? Following the consistency check framework, we measure the performance of forecasters in terms of the consistency of their predictions on different logically-related questions. We propose a new, general consistency metric based on arbitrage: for example, if a forecasting AI illogically predicts that both the Democratic and Republican parties have 60% probability of winning the 2024 US presidential election, an arbitrageur can trade against the forecaster's predictions and make a profit. We build an automated evaluation system that generates a set of base questions, instantiates consistency checks from these questions, elicits the predictions of the forecaster, and measures the consistency of the predictions. We then build a standard, proper-scoring-rule forecasting benchmark, and show that our (instantaneous) consistency metrics correlate with LLM forecasters' ground truth Brier scores (which are only known in the future). We also release a consistency benchmark that resolves in 2028, providing a long-term evaluation tool for forecasting.
Community HydroAgent: Formalizing Forecaster Expertise into Skill-Orchestrated Flood Forecasting Workflows
Politecnico di Milano（米兰理工大学）；China Three Gorges University（三峡大学）；Delft University of Technology（代尔夫特理工大学）；Qinhuangdao Hydrological Survey and Research Center of Hebei Province（河北省秦皇岛水文勘测研究中心）；Dalian University of Technology（大连理工大学）；TU Dresden University of Technology（德累斯顿工业大学）；Ningbo University（宁波大学）；China IPPR International Engineering Co., Ltd.（中国市政工程华北设计研究总院有限公司）；Hohai University（河海大学）；Nanjing University（南京大学）；Cornell University（康奈尔大学）；Tongji University（同济大学）；Zhejiang Institute of Hydraulics and Estuary（浙江省水利河口研究院）；Ghent University（根特大学）；University of British Columbia（不列颠哥伦比亚大学）；Beijing Normal University（北京师范大学）；Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工大学）
