Rundong Wang, Hongxin Wei, Bo An, Zhouyan Feng, Jun Yao
备注：
This paper was accepted by AAAI 2021 with the original title: Commission Fee is not Enough: A Hierarchical Reinforced Framework for Portfolio Management
Portfolio management via reinforcement learning is at the forefront of fintech research, which explores how to optimally reallocate a fund into different financial assets over the long term by trial-and-error. Existing methods are impractical since they usually assume each reallocation can be finished immediately and thus ignoring the price slippage as part of the trading cost. To address these issues, we propose a hierarchical reinforced stock trading system for portfolio management (HRPM). Concretely, we decompose the trading process into a hierarchy of portfolio management over trade execution and train the corresponding policies. The high-level policy gives portfolio weights at a lower frequency to maximize the long term profit and invokes the low-level policy to sell or buy the corresponding shares within a short time window at a higher frequency to minimize the trading cost. We train two levels of policies via pre-training scheme and iterative training scheme for data efficiency. Extensive experimental results in the U.S. market and the China market demonstrate that HRPM achieves significant improvement against many state-of-the-art approaches.
Pohang University of Science and Technology（浦项科技大学）；Seoul National University of Science and Technology（首尔国立科技大学）
Portfolio traders strive to identify dynamic portfolio allocation schemes so that their total budgets are efficiently allocated through the investment horizon. This study proposes a novel portfolio trading strategy in which an intelligent agent is trained to identify an optimal trading action by using deep Q-learning. We formulate a Markov decision process model for the portfolio trading process, and the model adopts a discrete combinatorial action space, determining the trading direction at prespecified trading size for each asset, to ensure practical applicability. Our novel portfolio tradin…
IITB-Monash Research Academy（IITB-莫纳什研究院）；IIT Bombay（印度理工学院孟买分校）；Monash University（莫纳什大学）
In stock trading, feature extraction and trading strategy design are the two important tasks to achieve long-term benefits using machine learning techniques. Several methods have been proposed to design trading strategy by acquiring trading signals to maximize the rewards. In the present paper the theory of deep reinforcement learning is applied for stock trading strategy and investment decisions to Indian markets. The experiments are performed systematically with three classical Deep Reinforcement Learning models Deep Q-Network, Double Deep Q-Network and Dueling Double Deep Q-Network on ten I…
Columbia University（哥伦比亚大学）；Nokia-Bell Labs（诺基亚贝尔实验室）
Stock trading strategy plays a crucial role in investment companies. However, it is challenging to obtain optimal strategy in the complex and dynamic stock market. We explore the potential of deep reinforcement learning to optimize stock trading strategy and thus maximize investment return. 30 stocks are selected as our trading stocks and their daily prices are used as the training and trading market environment. We train a deep reinforcement learning agent and obtain an adaptive trading strategy. The agent's performance is evaluated and compared with Dow Jones Industrial Average and the tradi…
A Comparative Analysis of Portfolio Optimization Using Mean-Variance, Hierarchical Risk Parity, and Reinforcement Learning Approaches on the Indian Stock Market
Stock portfolio optimization is the process of constant re-distribution of money to a pool of various stocks. In this paper, we will formulate the problem such that we can apply Reinforcement Learning for the task properly. To maintain a realistic assumption about the market, we will incorporate transaction cost and risk factor into the state as well. On top of that, we will apply various state-of-the-art Deep Reinforcement Learning algorithms for comparison. Since the action space is continuous, the realistic formulation were tested under a family of state-of-the-art continuous policy gradien…
The autonomous trading agent is one of the most actively studied areas of artificial intelligence to solve the capital market portfolio management problem. The two primary goals of the portfolio management problem are maximizing profit and restrainting risk. However, most approaches to this problem solely take account of maximizing returns. Therefore, this paper proposes a deep reinforcement learning based trading agent that can manage the portfolio considering not only profit maximization but also risk restraint. We also propose a new target policy to allow the trading agent to learn to prefe…
Institute for Development and Research in Banking Technology (IDRBT)（银行业技术发展与研究所（IDRBT））；University of Hyderabad（海得拉巴大学）
Explainable Artificial Intelligence (XAI) research gained prominence in recent years in response to the demand for greater transparency and trust in AI from the user communities. This is especially critical because AI is adopted in sensitive fields such as finance, medicine etc., where implications for society, ethics, and safety are immense. Following thorough systematic evaluations, work in XAI has primarily focused on Machine Learning (ML) for categorization, decision, or action. To the best of our knowledge, no work is reported that offers an Explainable Reinforcement Learning (XRL) method…
Schumpeter School of Business and Economics（熊彼特商业与经济学院）；University of Wuppertal（伍珀塔尔大学）
This paper proposes a Deep Reinforcement Learning algorithm for financial portfolio trading based on Deep Q-learning. The algorithm is capable of trading high-dimensional portfolios from cross-sectional datasets of any size which may include data gaps and non-unique history lengths in the assets. We sequentially set up environments by sampling one asset for each environment while rewarding investments with the resulting asset's return and cash reservation with the average return of the set of assets. This enforces the agent to strategically assign capital to assets that it predicts to perform …
Columbia University（哥伦比亚大学）；AI4Finance LLC.（AI4Finance有限责任公司）；Ion Media Networks（Ion媒体网络公司）；Imperial College（帝国理工学院）；New York University (Shanghai)（上海纽约大学）
As deep reinforcement learning (DRL) has been recognized as an effective approach in quantitative finance, getting hands-on experiences is attractive to beginners. However, to train a practical DRL trading agent that decides where to trade, at what price, and what quantity involves error-prone and arduous development and debugging. In this paper, we introduce a DRL library FinRL that facilitates beginners to expose themselves to quantitative finance and to develop their own stock trading strategies. Along with easily-reproducible tutorials, FinRL library allows users to streamline their own de…
Capital Fund Management（资本基金管理公司）；École Centrale-Supélec（中央理工-高等电力学院）；École Polytechnique（巴黎综合理工学院）
Can deep reinforcement learning algorithms be exploited as solvers for optimal trading strategies? The aim of this work is to test reinforcement learning algorithms on conceptually simple, but mathematically non-trivial, trading environments. The environments are chosen such that an optimal or close-to-optimal trading strategy is known. We study the deep deterministic policy gradient algorithm and show that such a reinforcement learning agent can successfully recover the essential features of the optimal trading strategies and achieve close-to-optimal rewards.
A Deep Reinforcement Learning Approach to Automated Stock Trading, using xLSTM Networks
Faculty of Computer Science and Engineering, Shahid Beheshti University（谢赫·贝赫什提大学计算机科学与工程学院）；Faculty of Electrical Engineering, Shahid Beheshti University（谢赫·贝赫什提大学电气工程学院）
Deep Reinforcement learning is a branch of unsupervised learning in which an agent learns to act based on environment state in order to maximize its total reward. Deep reinforcement learning provides good opportunity to model the complexity of portfolio choice in high-dimensional and data-driven environment by leveraging the powerful representation of deep neural networks. In this paper, we build a portfolio management system using direct deep reinforcement learning to make optimal portfolio choice periodically among S\&P500 underlying stocks by learning a good factor representation (as input)…
University of Electronic Science and Technology of China（电子科技大学）；Harbin Institute of Technology（哈尔滨工业大学）
It is a difficult task for both professional investors and individual traders continuously making profit in stock market. With the development of computer science and deep reinforcement learning, Buy\&Hold (B\&H) has been oversteped by many artificial intelligence trading algorithms. However, the information and process are not enough, which limit the performance of reinforcement learning algorithms. Thus, we propose a parallel-network continuous quantitative trading model with GARCH and PPO to enrich the basical deep reinforcement learning model, where the deep learning parallel network layer…
Unfair stock trading strategies have been shown to be one of the most negative perceptions that customers can have concerning trading and may result in long-term losses for a company. Investment banks usually place trading orders for multiple clients with the same target assets but different order sizes and diverse requirements such as time frame and risk aversion level, thereby total earning and individual earning cannot be optimized at the same time. Orders executed earlier would affect the market price level, so late execution usually means additional implementation cost. In this paper, we …
This paper is to explore the possibility to use alternative data and artificial intelligence techniques to trade stocks. The efficacy of the daily Twitter sentiment on predicting the stock return is examined using machine learning methods. Reinforcement learning(Q-learning) is applied to generate the optimal trading policy based on the sentiment signal. The predicting power of the sentiment signal is more significant if the stock price is driven by the expectation of the company growth and when the company has a major event that draws the public attention. The optimal trading strategy based on…
Deep Reinforcement Learning (DRL) algorithms can scale to previously intractable problems. The automation of profit generation in the stock market is possible using DRL, by combining the financial assets price "prediction" step and the "allocation" step of the portfolio in one unified process to produce fully autonomous systems capable of interacting with their environment to make optimal decisions through trial and error. This work represents a DRL model to generate profitable trades in the stock market, effectively overcoming the limitations of supervised learning approaches. We formulate th…
When AI Meets Finance (StockAgent): Large Language Model-based Stock Trading in Simulated Real-world Environments
University of Liverpool（利物浦大学）；Peking University（北京大学）；Shanghai University of Finance（上海金融学院）；Rutgers University（罗格斯大学）；University of Michigan（密歇根大学）；Northwestern University（西北大学）；New York University（纽约大学）；Xi’an Jiaotong-Liverpool University（西安交通大学利物浦大学）；New Jersey Institute of Technology（新泽西理工学院）
Just as war is sometimes fallaciously represented as a zero sum game -- when in fact war is a negative sum game - stock market trading, a positive sum game over time, is often erroneously represented as a zero sum game. This is called the "zero sum fallacy" -- the erroneous belief that one trader in a stock market exchange can only improve their position provided some other trader's position deteriorates. However, a positive sum game in absolute terms can be recast as a zero sum game in relative terms. Similarly it appears that negative sum games in absolute terms have been recast as zero sum …
Dr. B. C. Roy Engg. College（B.C.罗伊工程学院）；Bijoy Krishna Girls’ College（比乔伊·克里希纳女子学院）；National Institute of Technology Durgapur（杜尔加布尔国家理工学院）
Selection of proper stocks, before allocating investment ratios, is always a crucial task for the investors. Presence of many influencing factors in stock performance have motivated researchers to adopt various Artificial Intelligence (AI) techniques to make this challenging task easier. In this paper a novel fuzzy expert system model is proposed to evaluate and rank the stocks under Bombay Stock Exchange (BSE). Dempster-Shafer (DS) evidence theory is used for the first time to automatically generate the consequents of the fuzzy rule base to reduce the effort in knowledge base development of t…
This paper proposes a novel adaptive algorithm for the automated short-term trading of financial instrument. The algorithm adopts a semantic sentiment analysis technique to inspect the Twitter posts and to use them to predict the behaviour of the stock market. Indeed, the algorithm is specifically developed to take advantage of both the sentiment and the past values of a certain financial instrument in order to choose the best investment decision. This allows the algorithm to ensure the maximization of the obtainable profits by trading on the stock market. We have conducted an investment simul…
University of Toronto（多伦多大学）；Technion - Israel Institute of Technology（以色列理工学院）
A novel algorithm for actively trading stocks is presented. While traditional expert advice and "universal" algorithms (as well as standard technical trading heuristics) attempt to predict winners or trends, our approach relies on predictable statistical relations between all pairs of stocks in the market. Our empirical results on historical markets provide strong evidence that this type of technical trading can "beat the market" and moreover, can beat the best stock in the market. In doing so we utilize a new idea for smoothing critical parameters in the context of expert learning.
