arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.07355cs.LGcs.AIecon.GNq-fin.EC
收藏

预测竞技场:在真实世界预测市场中评估AI模型

Prediction Arena: Benchmarking AI Models on Real-World Prediction Markets

Arcada Labs;Harvard University(哈佛大学)

Jaden Zhang, Gardenia Liu, Oliver Johansson, Hileamlak Yitayew, Kamryn Ohly, Grace Li

AI总结:

本文提出预测竞技场,通过让AI模型在真实预测市场上自主交易来评估其预测准确性和决策能力。研究分析了不同模型在真实市场中的表现,揭示了预测准确性及正确预测的利用能力是关键因素,同时展示了平台设计对模型成功的影响。

摘要:

我们引入了预测竞技场,一种用于评估AI模型预测准确性和决策能力的基准,通过让模型在实时预测市场上自主交易来实现。不同于合成基准,预测竞技场测试模型在实际交易所执行交易的环境(Kalshi和Polymarket),提供无法被操纵或过拟合的客观真实结果。每个模型作为独立代理开始时有1万美元,每15-45分钟做出自主决策。在为期57天的纵向评估(2026年1月12日至3月9日)中,我们跟踪了两个群体:六个前沿模型在实时交易中(Cohort 1,完整周期)和四个下一代模型在模拟交易中(Cohort 2,3天初步阶段)。对于Cohort 1,最终Kalshi回报率从-16.0%到-30.8%。我们的分析发现了一个清晰的表现层次:初始预测准确性以及正确预测的利用能力是主要驱动因素,而研究量与结果无相关性。从平行Polymarket实时交易中出现了一个显著的跨平台对比:Cohort 1模型在Polymarket平均仅-1.1%,而在Kalshi为-22.6%,其中grok-4-20-checkpoint实现了71.4%的结算胜率,这是任何平台或群体中的最高水平。gemini-3.1-pro-preview(Cohort 2)在Kalshi上执行了零笔交易,但在3天内实现了6.02%的Polymarket回报率,这是任何模型在任一群体中的最佳回报率,证明了平台设计对哪些模型成功有深远影响。除了表现外,我们还分析了计算效率(token使用量、周期时间)、结算准确性、退出模式和市场偏好,提供了前沿模型在真实金融压力下行为的全面视角。

英文摘要:

We introduce Prediction Arena, a benchmark for evaluating AI models' predictive accuracy and decision-making by enabling them to trade autonomously on live prediction markets with real capital. Unlike synthetic benchmarks, Prediction Arena tests models in environments where trades execute on actual exchanges (Kalshi and Polymarket), providing objective ground truth that cannot be gamed or overfitted. Each model operates as an independent agent starting with $10,000, making autonomous decisions every 15-45 minutes. Over a 57-day longitudinal evaluation (January 12 to March 9, 2026), we track two cohorts: six frontier models in live trading (Cohort 1, full period) and four next-generation models in paper trading (Cohort 2, 3-day preliminary). For Cohort 1, final Kalshi returns range from -16.0% to -30.8%. Our analysis identifies a clear performance hierarchy: initial prediction accuracy and the ability to capitalize on correct predictions are the main drivers, while research volume shows no correlation with outcomes. A striking cross-platform contrast emerges from parallel Polymarket live trading: Cohort 1 models averaged only -1.1% on Polymarket vs. -22.6% on Kalshi, with grok-4-20-checkpoint achieving a 71.4% settlement win rate - the highest across any platform or cohort. gemini-3.1-pro-preview (Cohort 2), which executed zero trades on Kalshi, achieved +6.02% on Polymarket in 3 days - the best return of any model across either cohort - demonstrating that platform design has a profound effect on which models succeed. Beyond performance, we analyze computational efficiency (token usage, cycle time), settlement accuracy, exit patterns, and market preferences, providing a comprehensive view of how frontier models behave under real financial pressure.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑