The development of reinforced learning methods has extended application to many areas including algorithmic trading. In this paper trading on the stock exchange is interpreted into a game with a Markov property consisting of states, actions, and rewards. A system for trading the fixed volume of a financial instrument is proposed and experimentally tested; this is based on the asynchronous advantage actor-critic method with the use of several neural network architectures. The application of recurrent layers in this approach is investigated. The experiments were performed on real anonymized data. The best architecture demonstrated a trading strategy for the RTS Index futures (MOEX:RTSI) with a profitability of 66% per annum accounting for commission. The project source code is available via the following link: http://github.com/evgps/a3c_trading.
We summarize the fundamental issues at stake in algorithmic trading, and the progress made in this field over the last twenty years. We first present the key problems of algorithmic trading, describing the concepts of optimal execution, optimal placement, and price impact. We then discuss the most recent advances in algorithmic trading through the use of Machine Learning, discussing the use of Deep Learning, Reinforcement Learning, and Generative Adversarial Networks.
This editorial article partially informs the algorithmic trading community about launching of the new journal "Algorithmic Trading and Controls" (ATC). ATC is an online open-access journal that publishes novel works on algorithmic trading and its control methodologies. In this inaugural article, we discuss nine major challenges that contemporary Algo trading faces. There is nothing superstitiously magical about the number "nine," but so is any other one. Several of these challenges are at the strategy level, including for example, trading of illiquid securities or optimal portfolio execution. …
Instituto Superior de Economia e Gestão（里斯本大学经济与管理高等学院）；Universidade de Lisboa（里斯本大学）；University of New Haven（纽黑文大学）；Royal Bank of Canada（加拿大皇家银行）
Deep Reinforcement Learning (DRL) has become an appealing solution to algorithmic trading such as high frequency trading of stocks and cyptocurrencies. However, DRL have been shown to be susceptible to adversarial attacks. It follows that algorithmic trading DRL agents may also be compromised by such adversarial techniques, leading to policy manipulation. In this paper, we develop a threat model for deep trading policies, and propose two attack techniques for manipulating the performance of such policies at test-time. Furthermore, we demonstrate the effectiveness of the proposed attacks agains…
This scientific research paper presents an innovative approach based on deep reinforcement learning (DRL) to solve the algorithmic trading problem of determining the optimal trading position at any point in time during a trading activity in stock markets. It proposes a novel DRL trading strategy so as to maximise the resulting Sharpe ratio performance indicator on a broad range of stock markets. Denominated the Trading Deep Q-Network algorithm (TDQN), this new trading strategy is inspired from the popular DQN algorithm and significantly adapted to the specific algorithmic trading problem at ha…
We present the use of the fitted Q iteration in algorithmic trading. We show that the fitted Q iteration helps alleviate the dimension problem that the basic Q-learning algorithm faces in application to trading. Furthermore, we introduce a procedure including model fitting and data simulation to enrich training data as the lack of data is often a problem in realistic application. We experiment our method on both simulated environment that permits arbitrage opportunity and real-world environment by using prices of 450 stocks. In the former environment, the method performs well, implying that ou…
Price movement prediction has always been one of the traders' concerns in financial market trading. In order to increase their profit, they can analyze the historical data and predict the price movement. The large size of the data and complex relations between them lead us to use algorithmic trading and artificial intelligence. This paper aims to offer an approach using Twin-Delayed DDPG (TD3) and the daily close price in order to achieve a trading strategy in the stock and cryptocurrency markets. Unlike previous studies using a discrete action space reinforcement learning algorithm, the TD3 i…
Algorithmic stock trading has become a staple in today's financial market, the majority of trades being now fully automated. Deep Reinforcement Learning (DRL) agents proved to be to a force to be reckon with in many complex games like Chess and Go. We can look at the stock market historical price series and movements as a complex imperfect information environment in which we try to maximize return - profit and minimize risk. This paper reviews the progress made so far with deep reinforcement learning in the subdomain of AI in finance, more precisely, automated low-frequency quantitative stock …
In this paper, we propose stock trading based on the average tax basis. Recall that when selling stocks, capital gain should be taxed while capital loss can earn certain tax rebate. We learn the optimal trading strategies with and without considering taxes by reinforcement learning. The result shows that tax ignorance could induce more than 62% loss on the average portfolio returns, implying that taxes should be embedded in the environment of continuous stock trading on AI platforms.
The availability of data on digital traces is growing to unprecedented sizes, but inferring actionable knowledge from large-scale data is far from being trivial. This is especially important for computational finance, where digital traces of human behavior offer a great potential to drive trading strategies. We contribute to this by providing a consistent approach that integrates various datasources in the design of algorithmic traders. This allows us to derive insights into the principles behind the profitability of our trading strategies. We illustrate our approach through the analysis of Bi…
IITB-Monash Research Academy（IITB-莫纳什研究院）；IIT Bombay（印度理工学院孟买分校）；Monash University（莫纳什大学）
In stock trading, feature extraction and trading strategy design are the two important tasks to achieve long-term benefits using machine learning techniques. Several methods have been proposed to design trading strategy by acquiring trading signals to maximize the rewards. In the present paper the theory of deep reinforcement learning is applied for stock trading strategy and investment decisions to Indian markets. The experiments are performed systematically with three classical Deep Reinforcement Learning models Deep Q-Network, Double Deep Q-Network and Dueling Double Deep Q-Network on ten I…
MOT: A Mixture of Actors Reinforcement Learning Method by Optimal Transport for Algorithmic Trading
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Financial trading has been widely analyzed for decades with market participants and academics always looking for advanced methods to improve trading performance. Deep reinforcement learning (DRL), a recently reinvigorated method with significant success in multiple domains, still has to show its benefit in the financial markets. We use a deep Q-network (DQN) to design long-short trading strategies for futures contracts. The state space consists of volatility-normalized daily returns, with buying or selling being the reinforcement learning action and the total reward defined as the cumulative p…
Columbia University（哥伦比亚大学）；Nokia-Bell Labs（诺基亚贝尔实验室）
Stock trading strategy plays a crucial role in investment companies. However, it is challenging to obtain optimal strategy in the complex and dynamic stock market. We explore the potential of deep reinforcement learning to optimize stock trading strategy and thus maximize investment return. 30 stocks are selected as our trading stocks and their daily prices are used as the training and trading market environment. We train a deep reinforcement learning agent and obtain an adaptive trading strategy. The agent's performance is evaluated and compared with Dow Jones Industrial Average and the tradi…
We propose a novel portfolio trading system, which contains a feature preprocessing module and a trading module. The feature preprocessing module consists of various data processing operations, while in the trading part, we integrate the portfolio weight rebalance function with the trading algorithm and make the trading system fully automated and suitable for individual investors, holding a handful of stocks. The data preprocessing procedures are applied to remove the white noise in the raw data set and uncover the general pattern underlying the data set before the processed feature set is inp…
Financial trading aims to build profitable strategies to make wise investment decisions in the financial market. It has attracted interests in the machine learning community for a long time. This paper proposes to trade financial assets automatically using feature preprocessing skills and Recurrent Reinforcement Learning (RRL) algorithm. The strategy starts from technical indicators extracted from assets' market information. Then these technical indicators are preprocessed by Principal Component Analysis (PCA) and Discrete Wavelet Transform (DWT) and eventually inputted to the RRL algorithm to…
Financial markets are often driven by latent factors which traders cannot observe. Here, we address an algorithmic trading problem with collections of heterogeneous agents who aim to perform optimal execution or statistical arbitrage, where all agents filter the latent states of the world, and their trading actions have permanent and temporary price impact. This leads to a large stochastic game with heterogeneous agents. We solve the stochastic game by investigating its mean-field game (MFG) limit, with sub-populations of heterogeneous agents, and, using a convex analysis approach, we show tha…
In recent years, machine learning has become prevalent in numerous tasks, including algorithmic trading. Stock market traders utilize machine learning models to predict the market's behavior and execute an investment strategy accordingly. However, machine learning models have been shown to be susceptible to input manipulations called adversarial examples. Despite this risk, the trading domain remains largely unexplored in the context of adversarial learning. In this study, we present a realistic scenario in which an attacker influences algorithmic trading systems by using adversarial learning …
We propose a convex formulation for a trading system with the Conditional Value-at-Risk as a risk-adjusted performance measure under the notion of Direct Reinforcement Learning. Due to convexity, the proposed approach can uncover a lucrative trading policy in a "pure" online manner where it can interactively learn and update the policy without multi-epoch training and validation. We assess our proposed algorithm on a real financial market where it trades one of the largest US trust funds, SPDR, for three years. Numerical experiments demonstrate the algorithm's robustness in detecting central m…
A Deep Reinforcement Learning Approach to Automated Stock Trading, using xLSTM Networks
Faculty of Computer Science and Engineering, Shahid Beheshti University（谢赫·贝赫什提大学计算机科学与工程学院）；Faculty of Electrical Engineering, Shahid Beheshti University（谢赫·贝赫什提大学电气工程学院）
Modern Algorithmic Trading ("Algo") allows institutional investors and traders to liquidate or establish big security positions in a fully automated or low-touch manner. Most existing academic or industrial Algos focus on how to "slice" a big parent order into smaller child orders over a given time horizon. Few models rigorously tackle the actual placement of these child orders. Instead, placement is mostly done with a combination of empirical signals and heuristic decision processes. A self-contained, realistic, and fully functional Child Order Placement (COP) model may never exist due to all…
FLAG-Trader: Fusion LLM-Agent with Gradient-based Reinforcement Learning for Financial Trading
Harvard University（哈佛大学）；Stevens Institute of Technology（史蒂文斯理工学院）；Columbia University（哥伦比亚大学）；University of Minnesota（明尼苏达大学）；NVIDIA（英伟达公司）；Rensselaer Polytechnic Institute（伦斯勒理工学院）；TheFinAI；University of Manchester（曼彻斯特大学）
This paper examines the role of algorithmic trading in modern financial markets. Additionally, order types, characteristics, and special features of algorithmic trading are described under the lens provided by the large development of high frequency trading technology. Special order types are examined together with an intuitive description of the implied dynamics of the order book conditional to special orders (iceberg and hidden). The chapter provides an analysis of the transaction costs associated with trading activity and examines the most common trading strategy employed in the market. It …
We derive explicit recursive formulas for Target Close (TC) and Implementation Shortfall (IS) in the Almgren-Chriss framework. We explain how to compute the optimal starting and stopping times for IS and TC, respectively, given a minimum trading size. We also show how to add a minimum participation rate constraint (Percentage of Volume, PVol) for both TC and IS. We also study an alternative set of risk measures for the optimisation of algorithmic trading curves. We assume a self-similar process (e.g. Levy process, fractional Brownian motion or fractal process) and define a new risk measure, th…
We make several improvements to the mean-variance framework for optimal pre-trade algorithmic execution, by working with volume measures and generic price dynamics. Volume measures are the continuum analogies for discrete volume profiles commonly implemented in the execution industry. Execution then becomes an absolutely continuous measure over such a measure space, and its Radon-Nikodym derivative is commonly known as the Participation of Volume (PoV) function. The four impact cost components are all consistently built upon the PoV function. Some novel efforts are made for these linear impact…
Unfair stock trading strategies have been shown to be one of the most negative perceptions that customers can have concerning trading and may result in long-term losses for a company. Investment banks usually place trading orders for multiple clients with the same target assets but different order sizes and diverse requirements such as time frame and risk aversion level, thereby total earning and individual earning cannot be optimized at the same time. Orders executed earlier would affect the market price level, so late execution usually means additional implementation cost. In this paper, we …
We explore online inductive transfer learning, with a feature representation transfer from a radial basis function network formed of Gaussian mixture model hidden processing units to a direct, recurrent reinforcement learning agent. This agent is put to work in an experiment, trading the major spot market currency pairs, where we accurately account for transaction and funding costs. These sources of profit and loss, including the price trends that occur in the currency markets, are made available to the agent via a quadratic utility, who learns to target a position directly. We improve upon ea…
Despite the fact that an intraday market price distribution is not normal, the random walk model of price behaviour is as important for the understanding of basic principles of the market as the pendulum model is a starting point of many fundamental theories in physics. This model is a good zero order approximation for liquid fast moving markets where the queue position is less important than the price action. In this paper we present an exact solution for the cost of the static passive slice execution. It is shown, that if a price has a random walk behaviour, there is no optimal limit level f…
University of Electronic Science and Technology of China（电子科技大学）；Harbin Institute of Technology（哈尔滨工业大学）
It is a difficult task for both professional investors and individual traders continuously making profit in stock market. With the development of computer science and deep reinforcement learning, Buy\&Hold (B\&H) has been oversteped by many artificial intelligence trading algorithms. However, the information and process are not enough, which limit the performance of reinforcement learning algorithms. Thus, we propose a parallel-network continuous quantitative trading model with GARCH and PPO to enrich the basical deep reinforcement learning model, where the deep learning parallel network layer…
In recent years, a wide range of investment models have been created using artificial intelligence. Automatic trading by artificial intelligence can expand the range of trading methods, such as by conferring the ability to operate 24 hours a day and the ability to trade with high frequency. Automatic trading can also be expected to trade with more information than is available to humans if it can sufficiently consider past data. In this paper, we propose an investment agent based on a deep reinforcement learning model, which is an artificial intelligence model. The model considers the transact…
The autonomous trading agent is one of the most actively studied areas of artificial intelligence to solve the capital market portfolio management problem. The two primary goals of the portfolio management problem are maximizing profit and restrainting risk. However, most approaches to this problem solely take account of maximizing returns. Therefore, this paper proposes a deep reinforcement learning based trading agent that can manage the portfolio considering not only profit maximization but also risk restraint. We also propose a new target policy to allow the trading agent to learn to prefe…
Deep Reinforcement Learning (DRL) algorithms can scale to previously intractable problems. The automation of profit generation in the stock market is possible using DRL, by combining the financial assets price "prediction" step and the "allocation" step of the portfolio in one unified process to produce fully autonomous systems capable of interacting with their environment to make optimal decisions through trial and error. This work represents a DRL model to generate profitable trades in the stock market, effectively overcoming the limitations of supervised learning approaches. We formulate th…
Market manipulation is a strategy used by traders to alter the price of financial securities. One type of manipulation is based on the process of buying or selling assets by using several trading strategies, among them spoofing is a popular strategy and is considered illegal by market regulators. Some promising tools have been developed to detect manipulation, but cases can still be found in the markets. In this paper we model spoofing and pinging trading, two strategies that differ in the legal background but share the same elemental concept of market manipulation. We use a reinforcement lear…
University of Oxford（牛津大学）；Oxford-Man Institute of Quantitative Finance（牛津-曼氏量化金融研究所）
We adopt Deep Reinforcement Learning algorithms to design trading strategies for continuous futures contracts. Both discrete and continuous action spaces are considered and volatility scaling is incorporated to create reward functions which scale trade positions based on market volatility. We test our algorithms on the 50 most liquid futures contracts from 2011 to 2019, and investigate how performance varies across different asset classes including commodities, equity indices, fixed income and FX markets. We compare our algorithms against classical time series momentum strategies, and show tha…
Reinforcement learning (RL) techniques have shown great success in many challenging quantitative trading tasks, such as portfolio management and algorithmic trading. Especially, intraday trading is one of the most profitable and risky tasks because of the intraday behaviors of the financial market that reflect billions of rapidly fluctuating capitals. However, a vast majority of existing RL methods focus on the relatively low frequency trading scenarios (e.g., day-level) and fail to capture the fleeting intraday investment opportunities due to two major challenges: 1) how to effectively train …
We propose a microstructural modeling framework for studying optimal market making policies in a FIFO (first in first out) limit order book (LOB). In this context, the limit orders, market orders, and cancel orders arrivals in the LOB are modeled as Cox point processes with intensities that only depend on the state of the LOB. These are high-dimensional models which are realistic from a micro-structure point of view and have been recently developed in the literature. In this context, we consider a market maker who stands ready to buy and sell stock on a regular and continuous basis at a public…
School of Engineering（工程学院）；Zurich University of Applied Sciences（苏黎世应用科技大学）
This paper presents a Double Deep Q-Network algorithm for trading single assets, namely the E-mini S&P 500 continuous futures contract. We use a proven setup as the foundation for our environment with multiple extensions. The features of our trading agent are constantly being expanded to include additional assets such as commodities, resulting in four models. We also respond to environmental conditions, including costs and crises. Our trading agent is first trained for a specific time period and tested on new data and compared with the long-and-hold strategy as a benchmark (market). We analyze…
Columbia University（哥伦比亚大学）；University of Virginia（弗吉尼亚大学）；New York University Shanghai（纽约大学上海校区）
Deep reinforcement learning (DRL) has been envisioned to have a competitive edge in quantitative finance. However, there is a steep development curve for quantitative traders to obtain an agent that automatically positions to win in the market, namely \textit{to decide where to trade, at what price} and \textit{what quantity}, due to the error-prone programming and arduous debugging. In this paper, we present the first open-source framework \textit{FinRL} as a full pipeline to help quantitative traders overcome the steep learning curve. FinRL is featured with simplicity, applicability and exte…
