Trade-R1: 通过过程级推理验证弥合可验证奖励与随机环境
Trade-R1: Bridging Verifiable Rewards to Stochastic Environments via Process-Level Reasoning Verification
AI总结：
Trade-R1通过过程级推理验证，将可验证奖励与随机金融环境连接，减少奖励黑客问题，DSR在跨市场泛化中表现优异。
摘要：
强化学习（RL）已使大型语言模型（LLMs）在数学和编码等领域实现了显著的推理能力，其中可验证的奖励提供了清晰的信号。然而，将这一范式扩展到金融决策时受到市场随机性质的挑战：奖励虽然可验证，但本质上具有噪声，导致标准RL退化为奖励黑客。为此，我们提出了Trade-R1，一种模型训练框架，通过过程级推理验证将可验证奖励与随机环境连接起来。我们的关键创新是一种验证方法，将评估长时间金融文档上的推理问题转化为结构化的检索增强生成（RAG）任务。我们构建了一个三角一致性度量，评估检索到的证据、推理链和决策之间的成对对齐情况，以作为过滤噪声市场回报的有效性过滤器。我们探索了两种奖励整合策略：固定效应语义奖励（FSR）用于稳定对齐信号，动态效应语义奖励（DSR）用于耦合幅度优化。在不同国家资产选择上的实验表明，我们的范式减少了奖励黑客现象，DSR在跨市场泛化方面表现优异，同时保持了最高的推理一致性。
英文摘要：
Reinforcement Learning (RL) has enabled Large Language Models (LLMs) to achieve remarkable reasoning in domains like mathematics and coding, where verifiable rewards provide clear signals. However, extending this paradigm to financial decision is challenged by the market's stochastic nature: rewards are verifiable but inherently noisy, causing standard RL to degenerate into reward hacking. To address this, we propose Trade-R1, a model training framework that bridges verifiable rewards to stochastic environments via process-level reasoning verification. Our key innovation is a verification method that transforms the problem of evaluating reasoning over lengthy financial documents into a structured Retrieval-Augmented Generation (RAG) task. We construct a triangular consistency metric, assessing pairwise alignment between retrieved evidence, reasoning chains, and decisions to serve as a validity filter for noisy market returns. We explore two reward integration strategies: Fixed-effect Semantic Reward (FSR) for stable alignment signals, and Dynamic-effect Semantic Reward (DSR) for coupled magnitude optimization. Experiments on different country asset selection demonstrate that our paradigm reduces reward hacking, with DSR achieving superior cross-market generalization while maintaining the highest reasoning consistency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
