TraderBench: AI代理在对抗性资本市场中的鲁棒性如何?
TraderBench: How Robust Are AI Agents in Adversarial Capital Markets?
AI总结:
TraderBench通过结合静态任务与对抗性交易模拟,评估AI代理在动态市场中的鲁棒性,发现现有模型在加密交易中表现稳定但缺乏真实适应性。
摘要:
评估金融领域的AI代理面临两个关键挑战:静态基准需要昂贵的专家标注却忽略了现实交易中的动态决策过程,而基于LLM的评判者则在领域特定任务上引入了不可控的方差。我们介绍了TraderBench,一个解决这两个问题的基准。它结合了专家验证的静态任务(知识检索、分析推理)与对抗性交易模拟,该模拟仅根据实际表现(夏普比率、收益、回撤)评分,从而完全消除评判者的方差。该框架包含两个新的赛道:具有四个逐步市场操纵转换的加密交易,以及跨P&L准确性、希腊值和风险管理的期权衍生品评分。交易场景可以使用新市场数据刷新以防止基准污染。在约50个任务上评估13个模型(8B开源至前沿),我们发现:(1)13个模型中有8个在加密交易中得分约为33分,且在对抗性条件下变化小于1分,暴露了固定非适应性策略;(2)扩展思考有助于检索(+26分)但对交易无影响(+0.3加密,-0.1期权)。这些发现表明当前代理缺乏真正的市场适应性,凸显了金融领域基于绩效的评估需求。
英文摘要:
Evaluating AI agents in finance faces two key challenges: static benchmarks require costly expert annotation yet miss the dynamic decision-making central to real-world trading, while LLM-based judges introduce uncontrolled variance on domain-specific tasks. We introduce TraderBench, a benchmark that addresses both issues. It combines expert-verified static tasks (knowledge retrieval, analytical reasoning) with adversarial trading simulations scored purely on realized performance-Sharpe ratio, returns, and drawdown-eliminating judge variance entirely. The framework features two novel tracks: crypto trading with four progressive market-manipulation transforms, and options derivatives scoring across P&L accuracy, Greeks, and risk management. Trading scenarios can be refreshed with new market data to prevent benchmark contamination. Evaluating 13 models (8B open-source to frontier) on ~50 tasks, we find: (1) 8 of 13 models score ~33 on crypto with <1-point variation across adversarial conditions, exposing fixed non-adaptive strategies; (2) extended thinking helps retrieval (+26 points) but has zero impact on trading (+0.3 crypto, -0.1 options). These findings reveal that current agents lack genuine market adaptation, underscoring the need for performance-grounded evaluation in finance.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。