Many machine learning tasks involve subpopulation shift where the testing data distribution is a subpopulation of the training distribution. For such settings, a line of recent work has proposed the use of a variant of empirical risk minimization(ERM) known as distributionally robust optimization (DRO). In this work, we apply DRO to real, large-scale tasks with subpopulation shift, and observe that DRO performs relatively poorly, and moreover has severe instability. We identify one direct cause of this phenomenon: sensitivity of DRO to outliers in the datasets. To resolve this issue, we propose the framework of DORO, for Distributional and Outlier Robust Optimization. At the core of this approach is a refined risk function which prevents DRO from overfitting to potential outliers. We instantiate DORO for the Cressie-Read family of Rényi divergence, and delve into two specific instances of this family: CVaR and $χ^2$-DRO. We theoretically prove the effectiveness of the proposed method, and empirically show that DORO improves the performance and stability of DRO with experiments on large modern datasets, thereby positively addressing the open question raised by Hashimoto et al., 2018.
Distributionally Robust Federated Learning with Outlier Resilience
Division of Decision and Control Systems, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology（瑞典皇家理工学院电气工程与计算机科学系决策与控制系统 division）；Department of Control Science and Engineering, College of Electronics and Information Engineering, Tongji University（同济大学电子信息工程学院控制科学与工程系）；Mechanical Engineering and Material Science, Duke University（达特茅斯大学机械工程与材料科学系）
Outlier-Robust Sparse Estimation via Non-Convex Optimization
Brown University（布朗大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Duke University（杜克大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of California, San Diego（加利福尼亚大学圣迭戈分校）；University of Southern California（南加利福尼亚大学）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）
Understanding Contrastive Learning via Distributionally Robust Optimization
University of Science and Technology of China（中国科学技术大学）；Zhejiang University（浙江大学）；Institute of Dataspace, Hefei Comprehensive National Science Center（合肥综合性国家科学中心数据空间研究院）
Stochastic Approximation Approaches to Group Distributionally Robust Optimization and Beyond
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）
