AlphaForgeBench：用大型语言模型对端到端交易策略设计进行基准测试
AlphaForgeBench: Benchmarking End-to-End Trading Strategy Design with Large Language Models
AI总结：
提出AlphaForgeBench框架，将LLM从随机交易代理重新定义为量化研究员，通过生成可执行alpha因子和基于因子的交易策略，消除执行不稳定，实现可复现的金融推理评估。
摘要：
大型语言模型（LLMs）的快速发展催生了大量金融基准测试，从静态知识评估演变为交互式交易模拟。然而，现有的实时交易评估框架在很大程度上忽略了一个关键的失败模式：LLMs在金融不确定性下的序贯决策中表现出严重的行为不稳定性。通过大量实验，我们表明，当作为交易代理部署时，LLMs表现出极端的运行间方差，即使在确定性解码下也会产生不一致的动作序列，并且经常在相邻时间步产生不合理的动作翻转。我们将这些行为归因于LLMs的无状态自回归特性，它们缺乏对先前动作的持久记忆，以及它们对投资组合分配任务中连续到离散动作映射的敏感性。这些缺陷从根本上破坏了现有许多在线和离线交易基准的可靠性和可复现性。为了解决这些局限性，我们提出了AlphaForgeBench，一个原则性的评估框架，将LLMs重新定义为量化研究员而非随机交易代理。AlphaForgeBench不要求模型产生离散的交易动作，而是要求模型生成可执行的alpha因子，并基于金融知识构建基于因子的交易策略。这种范式将推理与执行机制解耦，实现了确定性和可复现的评估，同时与真实的量化研究工作流程保持一致。在多个最先进的LLM上进行的大量实验表明，AlphaForgeBench消除了执行引起的不稳定性，并为评估金融推理、策略制定和alpha发现提供了严格的基准。网页链接：https://finbrain-lab-hkustgz.github.io/AlphaForgeBench
英文摘要：
The rapid advancement of Large Language Models (LLMs) has led to a surge of financial benchmarks, evolving from static knowledge evaluation toward interactive trading simulations. However, existing frameworks for evaluating real-time trading largely overlook a critical failure mode: the severe behavioral instability of LLMs in sequential decision-making under financial uncertainty. Through extensive experiments, we show that when deployed as trading agents, LLMs exhibit extreme run-to-run variance, generate inconsistent action sequences even under deterministic decoding, and frequently produce irrational action flipping across adjacent time steps. We attribute these behaviors to the stateless autoregressive nature of LLMs, which lack persistent memory of prior actions, together with their sensitivity to continuous-to-discrete action mappings in portfolio allocation tasks. These deficiencies fundamentally undermine the reliability and reproducibility of many existing online and offline trading benchmarks. To address these limitations, we propose AlphaForgeBench, a principled evaluation framework that redefines LLMs as quantitative researchers rather than stochastic trading agents. Instead of producing discrete trading actions, AlphaForgeBench requires models to generate executable alpha factors and compose factor-based trading strategies grounded in financial knowledge. This paradigm decouples reasoning from execution mechanics, enabling deterministic and reproducible evaluation while remaining aligned with real-world quantitative research workflows. Extensive experiments across multiple state-of-the-art LLMs demonstrate that AlphaForgeBench eliminates execution-induced instability and provides a rigorous benchmark for evaluating financial reasoning, strategy formulation, and alpha discovery. Webpage at https://finbrain-lab-hkustgz.github.io/AlphaForgeBench
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
