It remains challenging to deploy existing risk-averse approaches to real-world applications. The reasons are multi-fold, including the lack of global optimality guarantee and the necessity of learning from long-term consecutive trajectories. Long-term consecutive trajectories are prone to involving visiting hazardous states, which is a major concern in the risk-averse setting. This paper proposes Short-Term VOlatility-controlled Policy Search (STOPS), a novel algorithm that solves risk-averse problems by learning from short-term trajectories instead of long-term trajectories. Short-term trajectories are more flexible to generate, and can avoid the danger of hazardous state visitations. By using an actor-critic scheme with an overparameterized two-layer neural network, our algorithm finds a globally optimal policy at a sublinear rate with proximal policy optimization and natural policy gradient, with effectiveness comparable to the state-of-the-art convergence rate of risk-neutral policy-search methods. The algorithm is evaluated on challenging Mujoco robot simulation tasks under the mean-variance evaluation metric. Both theoretical analysis and experimental results demonstrate a state-of-the-art level of STOPS' performance among existing risk-averse policy search methods.
Massachusetts Institute of Technology（麻省理工学院）；Worcester Polytechnic Institute（伍斯特理工学院）
Early classification algorithms help users react faster to their machine learning model's predictions. Early warning systems in hospitals, for example, let clinicians improve their patients' outcomes by accurately predicting infections. While early classification systems are advancing rapidly, a major gap remains: existing systems do not consider irregular time series, which have uneven and often-long gaps between their observations. Such series are notoriously pervasive in impactful domains like healthcare. We bridge this gap and study early classification of irregular time series, a new sett…
Politecnico di Milano（米兰理工大学）；Institute for Scientific Interchange Foundation（科学交流基金会）；Banca IMI（意大利投资银行）
In real-world decision-making problems, for instance in the fields of finance, robotics or autonomous driving, keeping uncertainty under control is as important as maximizing expected returns. Risk aversion has been addressed in the reinforcement learning literature through risk measures related to the variance of returns. However, in many cases, the risk is measured not only on a long-term perspective, but also on the step-wise rewards (e.g., in trading, to ensure the stability of the investment bank, it is essential to monitor the risk of portfolio positions on a daily basis). In this paper,…
Enhancing Deep Hedging of Options with Implied Volatility Surface Feedback Information
Department of Finance, HEC Montréal, Canada（蒙特利尔HEC金融系）；GERAD and Department of Decision Sciences, HEC Montréal, Canada（GERAD与蒙特利尔HEC决策科学系）；Concordia University, Department of Mathematics and Statistics, Canada（康科迪亚大学数学与统计学系）
Multi-Armed Bernoulli Bandits via Minimax Single-Arm Stopping
Shanghai Jiao Tong University（上海交通大学）；The University of Sydney（悉尼大学）；École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；Imperial College Business School（帝国理工学院商学院）
I model the belief formation and decision making processes of economic agents during a monetary policy regime change (an acceleration in the money supply) with a deep reinforcement learning algorithm in the AI literature. I show that when the money supply accelerates, the learning agents only adjust their actions, which include consumption and demand for real balance, after gathering learning experience for many periods. This delayed adjustments leads to low returns during transition periods. Once they start adjusting to the new environment, their welfare improves. Their changes in beliefs and…
University of California, San Diego（加州大学圣地亚哥分校）；Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Rewards and punishments in different forms are pervasive and present in a wide variety of decision-making scenarios. By observing the outcome of a sufficient number of repeated trials, one would gradually learn the value and usefulness of a particular policy or strategy. However, in a given environment, the outcomes resulting from different trials are subject to chance influence and variations. In learning about the usefulness of a given policy, significant costs are involved in systematically undertaking the sequential trials; therefore, in most learning episodes, one would wish to keep the c…
Data-Driven Merton's Strategies via Policy Randomization
Department of Applied Mathematics and School of Accounting and Finance, The Hong Kong Polytechnic University（应用数学系和会计与金融学院，香港理工大学）；Department of Systems Engineering and Engineering Management, The Chinese University of Hong Kong（系统工程与工程管理系，香港中文大学）；Department of Industrial Engineering and Operations Research & Data Science Institute, Columbia University（工业工程与运筹学系及数据科学研究所，哥伦比亚大学）
We develop confidence bounds that hold uniformly over time for off-policy evaluation in the contextual bandit setting. These confidence sequences are based on recent ideas from martingale analysis and are non-asymptotic, non-parametric, and valid at arbitrary stopping times. We provide algorithms for computing these confidence sequences that strike a good balance between computational and statistical efficiency. We empirically demonstrate the tightness of our approach in terms of failure probability and width and apply it to the "gated deployment" problem of safely upgrading a production conte…
Optimal stopping problems consider the question of deciding when to stop an observation-generating process in order to maximize a return. We examine the problem of simultaneously learning and planning in such domains, when data is collected directly from the environment. We propose GFSE, a simple and flexible model-free policy search method that reuses data for sample efficiency by leveraging problem structure. We bound the sample complexity of our approach to guarantee uniform convergence of policy value estimates, tightening existing PAC bounds to achieve logarithmic dependence on horizon le…
FedCau: A Proactive Stop Policy for Communication and Computation Efficient Federated Learning
School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology（瑞典皇家理工学院电气与计算机科学学院）；Ericsson（爱立信公司）；Uppsala University（乌普萨拉大学）
