发表机构
University of California, Riverside（加利福尼亚大学河滨分校）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
Backtrader-Bench是用于算法交易LLM智能体的基准框架，通过自生成MCQ解决评估难题，实验显示带工具智能体准确率显著高于无工具模型，还可生成强化学习训练语料以构建专用量化交易智能体。
AI 中文摘要
对大语言模型（LLM）编码智能体在算法交易任务中的评估存在困难，原因在于静态基准存在数据污染风险，且数值回测输出需要实际代码执行的真实结果。本文提出Backtrader-Bench，这一框架包含两个互补的流水线：确定性多项选择题（MCQ）流水线从五种交易策略、33种模板和三个难度层级的回测配置中生成问题，并配备独立校验器重新推导每个答案；生成器-求解器过滤流水线自主挖掘更难的问题：生成器编写经可执行代码验证的问题，将其转换为MCQ，并丢弃无工具求解器无需代码执行即可回答的问题。我们在30道精选问题组成的集合上评估了11种无工具模型（每种运行10次）和四种带工具配置，带工具增强的智能体单次准确率达90.0%（GPT-5.5和Opus 4.7），较最优无工具基准（10次运行平均73.0%）高出17个百分点；在38道单独挖掘的问题上，无工具准确率进一步下降，半数模型降至约随机概率水平（25%）。除评估外，该可扩展MCQ基础设施旨在生成强化学习的训练语料，最终目标是构建适用于量化交易工作流的专用智能体。
英文摘要
Evaluating LLM coding agents in algorithmic trading is difficult because static benchmarks risk data contamination and numerical backtest outputs require ground truth from actual code execution. We present Backtrader-Bench, a framework with two complementary pipelines. A deterministic multiple-choice question (MCQ) pipeline generates questions from backtest configurations across five trading strategies, 33 templates, and three difficulty tiers, with an independent checker that re-derives every answer. A generator-solver filtering pipeline autonomously mines harder questions: a generator writes questions verified by executable code, converts them to MCQs, and discards any that a no-tool solver can answer without code execution. We evaluate 11 models without tools (10 runs each) and four with-tools configurations on a 30-question curated set. Tool-augmented agents reach 90.0% accuracy in a single pass (GPT-5.5 and Opus 4.7), outperforming the best no-tools baselines (73.0%, averaged over 10 runs) by 17 percentage points. On 38 separately mined questions, no-tools accuracy drops further, with half the models falling to roughly random-chance level (25%). Beyond evaluation, the scalable MCQ infrastructure is designed to produce a training corpus for reinforcement learning, with the ultimate goal of building a specialized agent for quantitative trading workflows.
CommentsAccepted to the FinLLM Workshop at IJCAI 2026. Code and data: https://github.com/rzhao999/Backtrader-Bench
