Recent advancements in Large Language Models (LLMs) have empowered LLM agents to autonomously collect world information, over which to conduct reasoning to solve complex problems. Given this capability, increasing interests have been put into employing LLM agents for predicting international events, which can influence decision-making and shape policy development on an international scale. Despite such a growing interest, there is a lack of a rigorous benchmark of LLM agents' forecasting capability and reliability. To address this gap, we introduce MIRAI, a novel benchmark designed to systematically evaluate LLM agents as temporal forecasters in the context of international events. Our benchmark features an agentic environment with tools for accessing an extensive database of historical, structured events and textual news articles. We refine the GDELT event database with careful cleaning and parsing to curate a series of relational prediction tasks with varying forecasting horizons, assessing LLM agents' abilities from short-term to long-term forecasting. We further implement APIs to enable LLM agents to utilize different tools via a code-based interface. In summary, MIRAI comprehensively evaluates the agents' capabilities in three dimensions: 1) autonomously source and integrate critical information from large global databases; 2) write codes using domain-specific APIs and libraries for tool-use; and 3) jointly reason over historical knowledge from diverse formats and time to accurately predict future events. Through comprehensive benchmarking, we aim to establish a reliable framework for assessing the capabilities of LLM agents in forecasting international events, thereby contributing to the development of more accurate and trustworthy models for international relation analysis.
FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs
Fudan University（复旦大学）；Shanghai Innovation Institute（上海创新研究院）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学深圳分校）；National University of Singapore（新加坡国立大学）
Macroeconomic Forecasting with Large Language Models
School of Economics and Finance, Queen Mary University of London（伦敦大学女王学院经济与金融学院）；School of Business and Economics, Brandeis University（布兰迪大学商学院与经济学学院）
ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities
Forecasting Research Institute（预测研究所）；Federal Reserve Bank of Chicago（芝加哥联邦储备银行）；New York University（纽约大学）；University of California, Berkeley（加州大学伯克利分校）；University of Pennsylvania（宾夕法尼亚大学）
CivWorldBench: Evaluating Language World Models in Structured Multi-Agent Sandbox Worlds
The Chinese University of Hong Kong（香港中文大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；UniPat AI；Peking University（北京大学）；TME（腾讯音乐娱乐集团）；The Hong Kong University of Science and Technology（香港科技大学）
