FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance
Columbia University（哥伦比亚大学）；AI4Finance LLC.（AI4Finance有限责任公司）；Ion Media Networks（Ion媒体网络公司）；Imperial College（帝国理工学院）；New York University (Shanghai)（上海纽约大学）
Xiao-Yang Liu, Hongyang Yang, Qian Chen, Runjia Zhang, Liuqing Yang, Bowen Xiao, Christina Dan Wang
备注：
Deep Reinforcement Learning Workshop, 34th Conference on Neural Information Processing Systems (NeurIPS2020), Vancouver, Canada
As deep reinforcement learning (DRL) has been recognized as an effective approach in quantitative finance, getting hands-on experiences is attractive to beginners. However, to train a practical DRL trading agent that decides where to trade, at what price, and what quantity involves error-prone and arduous development and debugging. In this paper, we introduce a DRL library FinRL that facilitates beginners to expose themselves to quantitative finance and to develop their own stock trading strategies. Along with easily-reproducible tutorials, FinRL library allows users to streamline their own developments and to compare with existing schemes easily. Within FinRL, virtual environments are configured with stock market datasets, trading agents are trained with neural networks, and extensive backtesting is analyzed via trading performance. Moreover, it incorporates important trading constraints such as transaction cost, market liquidity and the investor's degree of risk-aversion. FinRL is featured with completeness, hands-on tutorial and reproducibility that favors beginners: (i) at multiple levels of time granularity, FinRL simulates trading environments across various stock markets, including NASDAQ-100, DJIA, S&P 500, HSI, SSE 50, and CSI 300; (ii) organized in a layered architecture with modular structure, FinRL provides fine-tuned state-of-the-art DRL algorithms (DQN, DDPG, PPO, SAC, A2C, TD3, etc.), commonly-used reward functions and standard evaluation baselines to alleviate the debugging workloads and promote the reproducibility, and (iii) being highly extendable, FinRL reserves a complete set of user-import interfaces. Furthermore, we incorporated three application demonstrations, namely single stock trading, multiple stock trading, and portfolio allocation. The FinRL library will be available on Github at link https://github.com/AI4Finance-LLC/FinRL-Library.
Columbia University（哥伦比亚大学）；University of Virginia（弗吉尼亚大学）；New York University Shanghai（纽约大学上海校区）
Deep reinforcement learning (DRL) has been envisioned to have a competitive edge in quantitative finance. However, there is a steep development curve for quantitative traders to obtain an agent that automatically positions to win in the market, namely \textit{to decide where to trade, at what price} and \textit{what quantity}, due to the error-prone programming and arduous debugging. In this paper, we present the first open-source framework \textit{FinRL} as a full pipeline to help quantitative traders overcome the steep learning curve. FinRL is featured with simplicity, applicability and exte…
Columbia University（哥伦比亚大学）；The University of Hong Kong（香港大学）；University of Virginia（弗吉尼亚大学）；Northwestern University（西北大学）；New York University (Shanghai)（上海纽约大学）；IDEA Research（IDEA研究院）
Deep reinforcement learning (DRL) has shown huge potentials in building financial market simulators recently. However, due to the highly complex and dynamic nature of real-world markets, raw historical financial data often involve large noise and may not reflect the future of markets, degrading the fidelity of DRL-based market simulators. Moreover, the accuracy of DRL-based market simulators heavily relies on numerous and diverse DRL agents, which increases demand for a universe of market environments and imposes a challenge on simulation speed. In this paper, we present a FinRL-Meta framework…
The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））
In recent years, many practitioners in quantitative finance have attempted to use Deep Reinforcement Learning (DRL) to build better quantitative trading (QT) strategies. Nevertheless, many existing studies fail to address several serious challenges, such as the non-stationary financial environment and the bias and variance trade-off when applying DRL in the real financial market. In this work, we proposed Safe-FinRL, a novel DRL-based high-freq stock trading strategy enhanced by the near-stationary financial environment and low bias and variance estimation. Our main contributions are twofold: …
School of Engineering（工程学院）；Zurich University of Applied Sciences（苏黎世应用科技大学）
This paper presents a Double Deep Q-Network algorithm for trading single assets, namely the E-mini S&P 500 continuous futures contract. We use a proven setup as the foundation for our environment with multiple extensions. The features of our trading agent are constantly being expanded to include additional assets such as commodities, resulting in four models. We also respond to environmental conditions, including costs and crises. Our trading agent is first trained for a specific time period and tested on new data and compared with the long-and-hold strategy as a benchmark (market). We analyze…
Dynamic Datasets and Market Environments for Financial Reinforcement Learning
Columbia University（哥伦比亚大学）；University of Virginia（弗吉尼亚大学）；Rice University（莱斯大学）；SIAT, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；New York University (Shanghai)（上海纽约大学）；Northwestern University（西北大学）；IDEA Research, International Digital Economy Academy（国际数字经济研究院IDEA研究院）
Algorithmic stock trading has become a staple in today's financial market, the majority of trades being now fully automated. Deep Reinforcement Learning (DRL) agents proved to be to a force to be reckon with in many complex games like Chess and Go. We can look at the stock market historical price series and movements as a complex imperfect information environment in which we try to maximize return - profit and minimize risk. This paper reviews the progress made so far with deep reinforcement learning in the subdomain of AI in finance, more precisely, automated low-frequency quantitative stock …
University of Liverpool（利物浦大学）；King’s College London（伦敦国王学院）；University of Warwick（华威大学）
Within the mathematical finance literature there is a rich catalogue of mathematical models for studying algorithmic trading problems -- such as market-making and optimal execution -- in limit order books. This paper introduces \mbtgym, a Python module that provides a suite of gym environments for training reinforcement learning (RL) agents to solve such model-based trading problems. The module is set up in an extensible way to allow the combination of different aspects of different models. It supports highly efficient implementations of vectorized environments to allow faster training of RL a…
Finance is a particularly difficult playground for deep reinforcement learning. However, establishing high-quality market environments and benchmarks for financial reinforcement learning is challenging due to three major factors, namely, low signal-to-noise ratio of financial data, survivorship bias of historical data, and model overfitting in the backtesting stage. In this paper, we present an openly accessible FinRL-Meta library that has been actively maintained by the AI4Finance community. First, following a DataOps paradigm, we will provide hundreds of market environments through an automa…
Recent years have witnessed the successful marriage of finance innovations and AI techniques in various finance applications including quantitative trading (QT). Despite great research efforts devoted to leveraging deep learning (DL) methods for building better QT strategies, existing studies still face serious challenges especially from the side of finance, such as the balance of risk and return, the resistance to extreme loss, and the interpretability of strategies, which limit the application of DL-based strategies in real-life financial markets. In this work, we propose AlphaStock, a novel…
Reinforcement learning (RL) is gaining attention by more and more researchers in quantitative finance as the agent-environment interaction framework is aligned with decision making process in many business problems. Most of the current financial applications using RL algorithms are based on model-free method, which still faces stability and adaptivity challenges. As lots of cutting-edge model-based reinforcement learning (MBRL) algorithms mature in applications such as video games or robotics, we design a new approach that leverages resistance and support (RS) level as regularization terms for…
Quantum Reinforcement Learning Trading Agent for Sector Rotation in the Taiwan Stock Market
Neuro Industry Research（神经产业研究）；Neuro Industry, Inc.（神经产业公司）；Department of Computer Science（计算机科学系）；The University of Alabama（阿拉巴马大学）；Industrial Technology Research Institute（工业技术研究 institutes）
When AI Meets Finance (StockAgent): Large Language Model-based Stock Trading in Simulated Real-world Environments
University of Liverpool（利物浦大学）；Peking University（北京大学）；Shanghai University of Finance（上海金融学院）；Rutgers University（罗格斯大学）；University of Michigan（密歇根大学）；Northwestern University（西北大学）；New York University（纽约大学）；Xi’an Jiaotong-Liverpool University（西安交通大学利物浦大学）；New Jersey Institute of Technology（新泽西理工学院）
The author proposes a finance trading strategy named Entropy Oriented Trading and apply thermodynamics on the strategy. The state variables are chosen so that the strategy satisfies the second law of thermodynamics. Using the law, the author proves that the rate of investment (ROI) of the strategy is equal to or more than the rate of price change.
We outline the idiosyncrasies of neural information processing and machine learning in quantitative finance. We also present some of the approaches we take towards solving the fundamental challenges we face.
Princeton University（普林斯顿大学）；Capital Fund Management（资本基金管理公司）；Imperial College（帝国理工学院）
We use a deep neural network to generate controllers for optimal trading on high frequency data. For the first time, a neural network learns the mapping between the preferences of the trader, i.e. risk aversion parameters, and the optimal controls. An important challenge in learning this mapping is that in intraday trading, trader's actions influence price dynamics in closed loop via the market impact. The exploration--exploitation tradeoff generated by the efficient execution is addressed by tuning the trader's preferences to ensure long enough trajectories are produced during the learning ph…
Newcastle University（纽卡斯尔大学）；University of Edinburgh（爱丁堡大学）
Although machine learning approaches have been widely used in the field of finance, to very successful degrees, these approaches remain bespoke to specific investigations and opaque in terms of explainability, comparability, and reproducibility. The primary objective of this research was to shed light upon this field by providing a generic methodology that was investigation-agnostic and interpretable to a financial markets practitioner, thus enhancing their efficiency, reducing barriers to entry, and increasing the reproducibility of experiments. The proposed methodology is showcased on two au…
The notion of "relativistic finance" became ingrained in public imagination and has been asserted in many mass-media reports. Yet, despite an observed drive of the most reputable Wall Street firms to establish their servers ever closer to the trading hubs, there is surprisingly little "hard" information related to relativistic delay of the trading orders. In this paper, the author uses modified M/M/G queue theory to describe propagation of the trading signal with finite velocity.
Georgia Institute of Technology（佐治亚理工学院）；J.P. Morgan AI Research（摩根大通人工智能研究院）
In finance, the weak form of the Efficient Market Hypothesis asserts that historic stock price and volume data cannot inform predictions of future prices. In this paper we show that, to the contrary, future intra-day stock prices could be predicted effectively until 2009. We demonstrate this using two different profitable machine learning-based trading strategies. However, the effectiveness of both approaches diminish over time, and neither of them are profitable after 2009. We present our implementation and results in detail for the period 2003-2017 and propose a novel idea: the use of such f…
University of Oxford（牛津大学）；Oxford-Man Institute of Quantitative Finance（牛津-曼量化金融研究院）
Cross-sectional strategies are a classical and popular trading style, with recent high performing variants incorporating sophisticated neural architectures. While these strategies have been applied successfully to data-rich settings involving mature assets with long histories, deploying them on instruments with limited samples generally produce over-fitted models with degraded performance. In this paper, we introduce Fused Encoder Networks -- a novel and hybrid parameter-sharing transfer ranking model. The model fuses information extracted using an encoder-attention module operated on a source…
In recent years, machine learning has become prevalent in numerous tasks, including algorithmic trading. Stock market traders utilize machine learning models to predict the market's behavior and execute an investment strategy accordingly. However, machine learning models have been shown to be susceptible to input manipulations called adversarial examples. Despite this risk, the trading domain remains largely unexplored in the context of adversarial learning. In this study, we present a realistic scenario in which an attacker influences algorithmic trading systems by using adversarial learning …
The purpose of this paper is to showcase trading strategies that give solutions to three difficult and intriguing problems in business finance, economics and statistics. The paper discusses trading strategies for both commodities and stocks but the main focus is on stock market trading at the New York Stock Exchange. Problem 1: Buy Low and Sell High. The buy low and sell high problem can be summarized like this: suppose the price of a commodity or stock fluctuates indefinitely, is there any explicit strategy for a trader to "ride the price waves" by buying low and selling high to eventually wi…
This chapter presents a history of international trade finance - the oldest domain of international finance - from its emergence in the Middle Ages up to today. We describe how the structure and governance of the global trade finance market changed over time and how trade credit instruments evolved. Trade finance products initially consisted of idiosyncratic assets issued by local merchants and bankers. The financing of international trade then became increasingly centralized and credit instruments were standardized through the diffusion of the local standards of consecutive leading trading ce…
Sources and Nonlinearity of High Volume Return Premium: An Empirical Study on the Differential Effects of Investor Identity versus Trading Intensity (2020-2024)
Flash crashes in financial markets have become increasingly important attracting attention from financial regulators, market makers as well as from the media and the broader audience. Systemic risk and propagation of shocks in financial markets is also a topic of great relevance that attracted increasing attention in recent years. In the present work we bridge the gap between these two topics with an in-depth investigation of the systemic risk structure of co-crashes in high frequency trading. We find that large co-crashes are systemic in their nature and differ from small crashes. We demonstr…
We utilize a fundamentally different model of trading costs to look at the effect of the opening of the Hong Kong Shanghai Connect that links the stock exchanges in the two cities, arguably the biggest event in international business and finance since Christopher Columbus set sail for India. We design a novel methodology that compensates for the lack of data on trading costs in China. We estimate trading costs across similar positions on the dual listed set of securities in Hong Kong and China, hoping to provide useful pieces of information to help scale 'The Great Wall of Chinese Securities T…
In this work we show that prediction uncertainty estimates gleaned from deep learning models can be useful inputs for influencing the relative allocation of risk capital across trades. In this way, consideration of uncertainty is important because it permits the scaling of investment size across trade opportunities in a principled and data-driven way. We showcase this insight with a prediction model and find clear outperformance based on a Sharpe ratio metric, relative to trading strategies that either do not take uncertainty into account, or that utilize an alternative market-based statistic …
Over the past few years, the futures market has been successfully developing in the North-West region. Futures markets are one of the most effective and liquid-visible trading mechanisms. A large number of buyers are forced to compete with each other and raise their prices. A large number of sellers make them reduce prices. Thus, the gap between the prices of offers of buyers and sellers is reduced due to high competition, and this is a good criterion for the liquidity of the market. This high degree of liquidity contributed to the fact that futures trading took such an important role in comme…
Silkswap is an automated market maker model designed for efficient stablecoin trading with minimal price impact. The original purpose of Silkswap is to facilitate the trading of fiat-pegged stablecoins with the stablecoin Silk, but it can be applied to any pair of stablecoins. The Silkswap invariant is a hybrid function that generates an asymmetric price impact curve. We present the derivation of the Silkswap model and its mathematical properties. We also compare different numerical methods used to solve the invariant equation. Finally, we compare our model with the well-known Curve Finance mo…
