MOT: A Mixture of Actors Reinforcement Learning Method by Optimal Transport for Algorithmic Trading
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Algorithmic trading refers to executing buy and sell orders for specific assets based on automatically identified trading opportunities. Strategies based on reinforcement learning (RL) have demonstrated remarkable capabilities in addressing algorithmic trading problems. However, the trading patterns differ among market conditions due to shifted distribution data. Ignoring multiple patterns in the data will undermine the performance of RL. In this paper, we propose MOT,which designs multiple actors with disentangled representation learning to model the different patterns of the market. Furthermore, we incorporate the Optimal Transport (OT) algorithm to allocate samples to the appropriate actor by introducing a regularization loss term. Additionally, we propose Pretrain Module to facilitate imitation learning by aligning the outputs of actors with expert strategy and better balance the exploration and exploitation of RL. Experimental results on real futures market data demonstrate that MOT exhibits excellent profit capabilities while balancing risks. Ablation studies validate the effectiveness of the components of MOT.
Shandong University of Finance and Economic（山东财经大学）
This scientific paper propose a novel portfolio optimization model using an improved deep reinforcement learning algorithm. The objective function of the optimization model is the weighted sum of the expectation and value at risk(VaR) of portfolio cumulative return. The proposed algorithm is based on actor-critic architecture, in which the main task of critical network is to learn the distribution of portfolio cumulative return using quantile regression, and actor network outputs the optimal portfolio weight by maximizing the objective function mentioned above. Meanwhile, we exploit a linear t…
A Deep Reinforcement Learning Approach to Automated Stock Trading, using xLSTM Networks
Faculty of Computer Science and Engineering, Shahid Beheshti University（谢赫·贝赫什提大学计算机科学与工程学院）；Faculty of Electrical Engineering, Shahid Beheshti University（谢赫·贝赫什提大学电气工程学院）
The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））
In recent years, many practitioners in quantitative finance have attempted to use Deep Reinforcement Learning (DRL) to build better quantitative trading (QT) strategies. Nevertheless, many existing studies fail to address several serious challenges, such as the non-stationary financial environment and the bias and variance trade-off when applying DRL in the real financial market. In this work, we proposed Safe-FinRL, a novel DRL-based high-freq stock trading strategy enhanced by the near-stationary financial environment and low bias and variance estimation. Our main contributions are twofold: …
Trading markets represent a real-world financial application to deploy reinforcement learning agents, however, they carry hard fundamental challenges such as high variance and costly exploration. Moreover, markets are inherently a multiagent domain composed of many actors taking actions and changing the environment. To tackle these type of scenarios agents need to exhibit certain characteristics such as risk-awareness, robustness to perturbations and low learning variance. We take those as building blocks and propose a family of four algorithms. First, we contribute with two algorithms that us…
Shanghai Jiao Tong University（上海交通大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Tencent Ads（腾讯广告）
Online Real-Time Bidding (RTB) is a complex auction game among which advertisers struggle to bid for ad impressions when a user request occurs. Considering display cost, Return on Investment (ROI), and other influential Key Performance Indicators (KPIs), large ad platforms try to balance the trade-off among various goals in dynamics. To address the challenge, we propose a Multi-ObjecTive Actor-Critics algorithm based on reinforcement learning (RL), named MoTiAC, for the problem of bidding optimization with various goals. In MoTiAC, objective-specific agents update the global network asynchrono…
In this paper, we present a new intrinsically motivated actor-critic algorithm for learning continuous motor skills directly from raw visual input. Our neural architecture is composed of a critic and an actor network. Both networks receive the hidden representation of a deep convolutional autoencoder which is trained to reconstruct the visual input, while the centre-most hidden representation is also optimized to estimate the state value. Separately, an ensemble of predictive world models generates, based on its learning progress, an intrinsic reward signal which is combined with the extrinsic…
Reinforcement learning (RL) techniques have shown great success in many challenging quantitative trading tasks, such as portfolio management and algorithmic trading. Especially, intraday trading is one of the most profitable and risky tasks because of the intraday behaviors of the financial market that reflect billions of rapidly fluctuating capitals. However, a vast majority of existing RL methods focus on the relatively low frequency trading scenarios (e.g., day-level) and fail to capture the fleeting intraday investment opportunities due to two major challenges: 1) how to effectively train …
Reinforcement learning (RL) is gaining attention by more and more researchers in quantitative finance as the agent-environment interaction framework is aligned with decision making process in many business problems. Most of the current financial applications using RL algorithms are based on model-free method, which still faces stability and adaptivity challenges. As lots of cutting-edge model-based reinforcement learning (MBRL) algorithms mature in applications such as video games or robotics, we design a new approach that leverages resistance and support (RS) level as regularization terms for…
University of Eastern Finland（东芬兰大学）；Shenzhen Technology University（深圳技术大学）
Intelligent robots provide a new insight into efficiency improvement in industrial and service scenarios to replace human labor. However, these scenarios include dense and dynamic obstacles that make motion planning of robots challenging. Traditional algorithms like A* can plan collision-free trajectories in static environment, but their performance degrades and computational cost increases steeply in dense and dynamic scenarios. Optimal-value reinforcement learning algorithms (RL) can address these problems but suffer slow speed and instability in network convergence. Network of policy gradie…
A widely-studied deep reinforcement learning (RL) technique known as Prioritized Experience Replay (PER) allows agents to learn from transitions sampled with non-uniform probability proportional to their temporal-difference (TD) error. Although it has been shown that PER is one of the most crucial components for the overall performance of deep RL methods in discrete action domains, many empirical studies indicate that it considerably underperforms actor-critic algorithms in continuous control. We theoretically show that actor networks cannot be effectively trained with transitions that have la…
University of Liverpool（利物浦大学）；King’s College London（伦敦国王学院）；University of Warwick（华威大学）
Within the mathematical finance literature there is a rich catalogue of mathematical models for studying algorithmic trading problems -- such as market-making and optimal execution -- in limit order books. This paper introduces \mbtgym, a Python module that provides a suite of gym environments for training reinforcement learning (RL) agents to solve such model-based trading problems. The module is set up in an extensible way to allow the combination of different aspects of different models. It supports highly efficient implementations of vectorized environments to allow faster training of RL a…
Despite definite success in deep reinforcement learning problems, actor-critic algorithms are still confronted with sample inefficiency in complex environments, particularly in tasks where efficient exploration is a bottleneck. These methods consider a policy (the actor) and a value function (the critic) whose respective losses are built using different motivations and approaches. This paper introduces a third protagonist: the adversary. While the adversary mimics the actor by minimizing the KL-divergence between their respective action distributions, the actor, in addition to learning to solv…
Meta-Learning Reinforcement Learning for Crypto-Return Prediction
Sichuan University（四川大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Northwestern University（西北大学）；Huazhong University of Science and Technology（华中科技大学）；Queen’s University（女王大学）；University of Toronto（多伦多大学）；TrueNorth；Tsinghua University（清华大学）；Emory University（埃默里大学）；University of Macau（澳门大学）
Institute for Systems Research, University of Maryland（马里兰大学系统研究所）；Astrome Technologies Pvt Ltd（Astrome科技私人有限公司）；Indian Institute of Science（印度科学学院）
We propose a novel actor-critic algorithm with guaranteed convergence to an optimal policy for a discounted reward Markov decision process. The actor incorporates a descent direction that is motivated by the solution of a certain non-linear optimization problem. We also discuss an extension to incorporate function approximation and demonstrate the practicality of our algorithms on a network routing application.
Deep Reinforcement Learning (DRL) algorithms for continuous action spaces are known to be brittle toward hyperparameters as well as \cut{being}sample inefficient. Soft Actor Critic (SAC) proposes an off-policy deep actor critic algorithm within the maximum entropy RL framework which offers greater stability and empirical gains. The choice of policy distribution, a factored Gaussian, is motivated by \cut{chosen due}its easy re-parametrization rather than its modeling power. We introduce Normalizing Flow policies within the SAC framework that learn more expressive classes of policies than simple…
We develop an approach for solving time-consistent risk-sensitive stochastic optimization problems using model-free reinforcement learning (RL). Specifically, we assume agents assess the risk of a sequence of random variables using dynamic convex risk measures. We employ a time-consistent dynamic programming principle to determine the value of a particular policy, and develop policy gradient update rules that aid in obtaining optimal policies. We further develop an actor-critic style algorithm using neural networks to optimize over policies. Finally, we demonstrate the performance and flexibil…
We generalize the existing principle of the maximum Shannon entropy in reinforcement learning (RL) to weighted entropy by characterizing the state-action pairs with some qualitative weights, which can be connected with prior knowledge, experience replay, and evolution process of the policy. We propose an algorithm motivated for self-balancing exploration with the introduced weight function, which leads to state-of-the-art performance on Mujoco tasks despite its simplicity in implementation.
Most approaches to visual scene analysis have emphasised parallel processing of the image elements. However, one area in which the sequential nature of vision is apparent, is that of segmenting multiple, potentially similar and partially occluded objects in a scene. In this work, we revisit the recurrent formulation of this challenging problem in the context of reinforcement learning. Motivated by the limitations of the global max-matching assignment of the ground-truth segments to the recurrent states, we develop an actor-critic approach in which the actor recurrently predicts one instance ma…
Looped Actor: Depth-Recurrent Reasoning Models for Reinforcement Learning
ELLIS Institute Tübingen（ELLIS研究所蒂宾根）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Tübingen AI Center（蒂宾根人工智能中心）；Liquid AI；CSAIL, MIT（麻省理工学院计算机科学与人工智能实验室）；Case Western Reserve University（凯斯西储大学）
Asset allocation (or portfolio management) is the task of determining how to optimally allocate funds of a finite budget into a range of financial instruments/assets such as stocks. This study investigated the performance of reinforcement learning (RL) when applied to portfolio management using model-free deep RL agents. We trained several RL agents on real-world stock prices to learn how to perform asset allocation. We compared the performance of these RL agents against some baseline agents. We also compared the RL agents among themselves to understand which classes of agents performed better…
We study a simple exchange model in which price is fixed and the amount of a good transferred between actors depends only on the actors' respective budgets and the existence of a link between transacting actors. The model induces a simply-connected but possibly multi-component bipartite graph. A trading session on a fixed graph consists of a sequence of exchanges between connected buyers and sellers until no more exchanges are possible. We deem a trading session "feasible" if all of the buyers satisfy their respective demands. If all trading sessions are feasible the graph is deemed "successfu…
The development of reinforced learning methods has extended application to many areas including algorithmic trading. In this paper trading on the stock exchange is interpreted into a game with a Markov property consisting of states, actions, and rewards. A system for trading the fixed volume of a financial instrument is proposed and experimentally tested; this is based on the asynchronous advantage actor-critic method with the use of several neural network architectures. The application of recurrent layers in this approach is investigated. The experiments were performed on real anonymized data…
We study a single-period optimal transport problem on $\mathbb{R}^2$ with a covariance-type cost function $c(x,y) = (x_1-y_1)(x_2-y_2)$ and a backward martingale constraint. We show that a transport plan $γ$ is optimal if and only if there is a maximal monotone set $G$ that supports the $x$-marginal of $γ$ and such that $c(x,y) = \min_{z\in G}c(z,y)$ for every $(x,y)$ in the support of $γ$. We obtain sharp regularity conditions for the uniqueness of an optimal plan and for its representation in terms of a map. Our study is motivated by a variant of the classical Kyle model of insider trading f…
We consider a limit order book, where buyers and sellers register to trade a security at specific prices. The largest price buyers on the book are willing to offer is called the market bid price, and the smallest price sellers on the book are willing to accept is called the market ask price. Market ask price is always greater than market bid price, and these prices move upwards and downwards due to new arrivals, market trades, and cancellations. We model these two price processes as "bouncing geometric Brownian motions (GBMs)", which are defined as exponentials of two mutually reflected Browni…
We develop a fundamentally different stochastic dynamic programming model of trading costs. Built on a strong theoretical foundation, our model provides insights to market participants by splitting the overall move of the security price during the duration of an order into the Market Impact (price move caused by their actions) and Market Timing (price move caused by everyone else) components. We derive formulations of this model under different laws of motion of the security prices, starting with a simple benchmark scenario and extending this to include multiple sources of uncertainty, liquidi…
We revisit optimal execution of an active portfolio in the presence of slippage (aka linear, proportional, or absolute-value) costs. Market efficiency implies a close balance between active alphas and trading costs, so even small changes to trading optimization can make a big difference. It has been observed for some time that optimal trading involves a pattern of a no-trade zone with width $Δ$ increasing with slippage cost parameter $c$. In a setting of a reasonably stable (non-stochastic) forecast of future returns and a quadratic risk aversion, it is shown that $Δ\sim c^{1/2}$, which differ…
We derive explicit recursive formulas for Target Close (TC) and Implementation Shortfall (IS) in the Almgren-Chriss framework. We explain how to compute the optimal starting and stopping times for IS and TC, respectively, given a minimum trading size. We also show how to add a minimum participation rate constraint (Percentage of Volume, PVol) for both TC and IS. We also study an alternative set of risk measures for the optimisation of algorithmic trading curves. We assume a self-similar process (e.g. Levy process, fractional Brownian motion or fractal process) and define a new risk measure, th…
We study the optimal timing strategies for trading a mean-reverting price process with afinite deadline to enter and a separate finite deadline to exit the market. The price process is modeled by a diffusion with an affine drift that encapsulates a number of well-known models,including the Ornstein-Uhlenbeck (OU) model, Cox-Ingersoll-Ross (CIR) model, Jacobi model,and inhomogeneous geometric Brownian motion (IGBM) model.We analyze three types of trading strategies: (i) the long-short (long to open, short to close) strategy; (ii) the short-long(short to open, long to close) strategy, and (iii) …
The purpose of this paper is to showcase trading strategies that give solutions to three difficult and intriguing problems in business finance, economics and statistics. The paper discusses trading strategies for both commodities and stocks but the main focus is on stock market trading at the New York Stock Exchange. Problem 1: Buy Low and Sell High. The buy low and sell high problem can be summarized like this: suppose the price of a commodity or stock fluctuates indefinitely, is there any explicit strategy for a trader to "ride the price waves" by buying low and selling high to eventually wi…
Motivated by the industry practice of pairs trading, we study the optimal timing strategies for trading a mean-reverting price spread. An optimal double stopping problem is formulated to analyze the timing to start and subsequently liquidate the position subject to transaction costs. Modeling the price spread by an Ornstein-Uhlenbeck process, we apply a probabilistic methodology and rigorously derive the optimal price intervals for market entry and exit. As an extension, we incorporate a stop-loss constraint to limit the maximum loss. We show that the entry region is characterized by a bounded…
Motivated by how transaction amount constrain trading volume and price volatility in stock market, we, in this paper, study the relation between volume and price if amount of transaction is given. We find that accumulative trading volume gradually emerges a kurtosis near the price mean value over a trading price range when it takes a longer trading time, regardless of actual price fluctuation path, time series, or total transaction volume in the time interval. To explain the volume-price behavior, we, in terms of physics, propose a transaction energy hypothesis, derive a time-independent trans…
Motivated by the literature on investment flows and optimal trading, we examine intraday predictability in the cross-section of stock returns. We find a striking pattern of return continuation at half-hour intervals that are exact multiples of a trading day, and this effect lasts for at least 40 trading days. Volume, order imbalance, volatility, and bid-ask spreads exhibit similar patterns, but do not explain the return patterns. We also show that short-term return reversal is driven by temporary liquidity imbalances lasting less than an hour and bid-ask bounce. Timing trades can reduce execut…
