arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2208.06244cs.CEcs.LG
收藏

一种强化学习最优执行的模块化框架

A Modular Framework for Reinforcement Learning Optimal Execution

Zurich University of Applied Sciences(苏黎世应用科学大学)

Fernando de Meer Pardo, Christoph Auth, Florin Dascalu

摘要:

在这篇文章中,我们开发了一个模块化框架,用于将强化学习应用于最优交易执行问题。该框架的设计考虑了灵活性,以简化不同模拟设置的实施。而不是专注于代理和优化方法,我们专注于环境,并分解必要的要求,以模拟强化学习框架下的最佳交易执行,如数据预处理,观察的构建,动作处理,子订单执行,模拟基准,奖励计算等。探讨它们各自的实现所面临的困难以及它们之间的相互作用,并讨论每个组件在模拟中引起的不同现象,突出模拟与真实市场行为之间的差异。我们通过一个设置展示了我们的模块化实现,该设置遵循时间加权平均价格(TWAP)订单提交时间表,允许代理专门下限价订单,通过迭代限价订单簿(LOB)的快照来模拟它们的执行,并计算奖励作为TWAP基准算法遵循相同时间表实现的价格的改进。我们还开发了评估程序,其中包括在训练范围的间隔内对给定代理进行迭代再训练和评估,模仿代理在不断重新训练时的行为,因为新的市场数据变得可用,并模仿算法提供商在当前监管框架下必须执行的监控实践。

英文摘要:

In this article, we develop a modular framework for the application of Reinforcement Learning to the problem of Optimal Trade Execution. The framework is designed with flexibility in mind, in order to ease the implementation of different simulation setups. Rather than focusing on agents and optimization methods, we focus on the environment and break down the necessary requirements to simulate an Optimal Trade Execution under a Reinforcement Learning framework such as data pre-processing, construction of observations, action processing, child order execution, simulation of benchmarks, reward calculations etc. We give examples of each component, explore the difficulties their individual implementations \& the interactions between them entail, and discuss the different phenomena that each component induces in the simulation, highlighting the divergences between the simulation and the behavior of a real market. We showcase our modular implementation through a setup that, following a Time-Weighted Average Price (TWAP) order submission schedule, allows the agent to exclusively place limit orders, simulates their execution via iterating over snapshots of the Limit Order Book (LOB), and calculates rewards as the \$ improvement over the price achieved by a TWAP benchmark algorithm following the same schedule. We also develop evaluation procedures that incorporate iterative re-training and evaluation of a given agent over intervals of a training horizon, mimicking how an agent may behave when being continuously retrained as new market data becomes available and emulating the monitoring practices that algorithm providers are bound to perform under current regulatory frameworks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑