你的离线策略并不可靠：面向序列投资组合优化的双层强化学习
Your Offline Policy is Not Trustworthy: Bilevel Reinforcement Learning for Sequential Portfolio Optimization
AI总结：
针对传统离线强化学习在序列投资组合优化中泛化性差、易受市场非平稳性影响的问题，提出采用双层框架与最坏情况时序差分的MetaTrader方法，在公开股票数据集上表现优于现有方法。
摘要：
强化学习（RL）在序列投资组合优化任务（如股票交易）中展现出巨大潜力，这类任务的目标是利用历史数据在最小化风险的同时最大化累计收益。然而，传统RL方法生成的策略往往只是记住了固定数据集中最优却不切实际的买卖行为。这类离线策略泛化性较差，因为它们未考虑市场的非平稳特性。我们提出的方法MetaTrader将投资组合优化构建为一类新型的部分离线RL问题，并做出两项技术贡献。首先，MetaTrader采用双层学习框架，显式训练RL智能体以同时提升在原始数据集上的域内收益，以及在原始金融数据的各类变换下的域外性能。其次，我们的方法引入了一种新的时序差分（TD）方法，该方法从一批变换后的TD目标中近似最坏情况的TD估计，解决了在离线数据有限的场景下尤为突出的价值高估问题。我们在两个公开股票数据集上的实验结果表明，MetaTrader的性能优于现有方法，包括基于RL的方法和传统的股票预测模型。
英文摘要：
Reinforcement learning (RL) has shown significant promise for sequential portfolio optimization tasks, such as stock trading, where the objective is to maximize cumulative returns while minimizing risks using historical data. However, traditional RL approaches often produce policies that merely memorize the optimal yet impractical buying and selling behaviors within the fixed dataset. These offline policies are less generalizable as they fail to account for the non-stationary nature of the market. Our approach, MetaTrader, frames portfolio optimization as a new type of partial-offline RL problem and makes two technical contributions. First, MetaTrader employs a bilevel learning framework that explicitly trains the RL agent to improve both in-domain profits on the original dataset and out-of-domain performance across diverse transformations of the raw financial data. Second, our approach incorporates a new temporal difference (TD) method that approximates worst-case TD estimates from a batch of transformed TD targets, addressing the value overestimation issue that is particularly challenging in scenarios with limited offline data. Our empirical results on two public stock datasets show that MetaTrader outperforms existing methods, including both RL-based approaches and traditional stock prediction models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
