WILDS: A Benchmark of in-the-Wild Distribution Shifts
Microsoft（微软）
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena … 展开作者
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, Percy Liang
Distribution shifts -- where the training distribution differs from the test distribution -- can substantially degrade the accuracy of machine learning (ML) systems deployed in the wild. Despite their ubiquity in the real-world deployments, these distribution shifts are under-represented in the datasets widely used in the ML community today. To address this gap, we present WILDS, a curated benchmark of 10 datasets reflecting a diverse range of distribution shifts that naturally arise in real-world applications, such as shifts across hospitals for tumor identification; across camera traps for wildlife monitoring; and across time and location in satellite imaging and poverty mapping. On each dataset, we show that standard training yields substantially lower out-of-distribution than in-distribution performance. This gap remains even with models trained by existing methods for tackling distribution shifts, underscoring the need for new methods for training models that are more robust to the types of distribution shifts that arise in practice. To facilitate method development, we provide an open-source package that automates dataset loading, contains default model architectures and hyperparameters, and standardizes evaluations. Code and leaderboards are available at https://wilds.stanford.edu.
Feed Two Birds with One Scone: Exploiting Wild Data for Both Out-of-Distribution Generalization and Detection
Institute for Foundations of Data Science, University of Wisconsin, Madison（数据科学基础研究院，威斯康星大学麦迪逊分校）；Department of Computer Sciences, University of Wisconsin, Madison（计算机科学系，威斯康星大学麦迪逊分校）；Department of Electrical and Computer Engineering, University of Wisconsin, Madison（电气与计算机工程系，威斯康星大学麦迪逊分校）
WildRoadBench: A Wild Aerial Road-Damage Grounding Benchmark for Vision-Language Models and Autonomous Agents
University of Electronic Science and Technology of China（电子科技大学）；National University of Singapore（新加坡国立大学）；De Artificial Intelligence Lab（德人工智能实验室）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；University of Science and Technology of China（中国科学技术大学）
MassSpecGym in the Wild: Uncovering and Correcting Evaluation Pitfalls in AI-Driven Molecule Discovery
Massachusetts Institute of Technology（麻省理工学院）；Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague（捷克信息学、机器人学与自动化捷克技术大学）；Enveda Biosciences（Enveda 生物科技）；Tufts University（塔夫茨大学）
CLUTCH: Contextualized Language model for Unlocking Text-Conditioned Hand motion modelling in the wild
Technical University of Darmstadt（德累斯顿技术大学）；Max-Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Tuebingen（图宾根大学）；Tubingen AI Center（图宾根人工智能中心）；Max Planck Institute for Informatics（马克斯·普朗克信息研究所）
The Swiss AI Lab, IDSIA, University of Lugano (USI) & SUPSI（瑞士人工智能实验室 IDSIA，卢加诺大学 (USI) 与南瑞士应用科学与艺术大学 (SUPSI)）；King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学 (KAUST)）
PTSD in the Wild: A Video Database for Studying Post-Traumatic Stress Disorder Recognition in Unconstrained Environments
University Paris-Est Créteil（巴黎东克雷泰伊大学）；University Paris City（巴黎城市大学）；Istanbul Technical University（伊斯坦布尔理工大学）；National Institute of Technology Warangal（瓦朗加尔国家技术学院）；The National Engineering School of Sfax（斯法克斯国家工程师学院）；French National Centre for Scientific Research（法国国家科学研究中心）
