Despite the remarkable performance that modern deep neural networks have achieved on independent and identically distributed (I.I.D.) data, they can crash under distribution shifts. Most current evaluation methods for domain generalization (DG) adopt the leave-one-out strategy as a compromise on the limited number of domains. We propose a large-scale benchmark with extensive labeled domains named NICO++ along with more rational evaluation methods for comprehensively evaluating DG algorithms. To evaluate DG datasets, we propose two metrics to quantify covariate shift and concept shift, respectively. Two novel generalization bounds from the perspective of data construction are proposed to prove that limited concept shift and significant covariate shift favor the evaluation capability for generalization. Through extensive experiments, NICO++ shows its superior evaluation capability compared with current DG datasets and its contribution in alleviating unfairness caused by the leak of oracle knowledge in model selection.
Beijing University of Posts and Telecommunications（北京邮电大学）；China Mobile Research Institute（中国移动研究院）
Recently, out-of-distribution (OOD) generalization has attracted attention to the robustness and generalization ability of deep learning based models, and accordingly, many strategies have been made to address different aspects related to this issue. However, most existing algorithms for OOD generalization are complicated and specifically designed for certain dataset. To alleviate this problem, nicochallenge-2022 provides NICO++, a large-scale dataset with diverse context information. In this paper, based on systematic analysis of different schemes on NICO++ dataset, we propose a simple but ef…
Benchmarking Domain Generalization Algorithms in Computational Pathology
Tissue Image Analytics centre, Department of Computer Science（组织机构：组织机构，计算机科学系）；Histofy Ltd（Histofy有限公司）；ICMR-National Institute of Pathology（ICMR国家病理研究所）
DispBench: Benchmarking Disparity Estimation to Synthetic Corruptions
Data and Web Science Group, University of Mannheim（曼海姆大学数据与网络科学小组）；Max-Planck-Institute for Informatics, Saarland Informatics Campus（马克斯·普朗克研究所信息学研究所萨尔兰信息学校区）
本文提出DispBench基准测试工具，系统评估视差估计方法在对抗攻击和2D Common Corruptions等合成损坏下的鲁棒性，并通过迄今最广泛的分析揭示准确率、可靠性与泛化能力之间的关键相关性。
SUB: Benchmarking CBM Generalization via Synthetic Attribute Substitutions
Technical University of Munich, Helmholtz Munich, Munich Center for Machine Learning (MCML)（慕尼黑技术大学、亥姆霍兹慕尼黑、慕尼黑机器学习中心）；LTCI, Télécom Paris, Institut Polytechnique de Paris（LTCI、巴黎电信、巴黎理工学院）
University of Western Australia（西澳大学）；Murdoch University（莫道克大学）
In stereo vision, self-similar or bland regions can make it difficult to match patches between two images. Active stereo-based methods mitigate this problem by projecting a pseudo-random pattern on the scene so that each patch of an image pair can be identified without ambiguity. However, the projected pattern significantly alters the appearance of the image. If this pattern acts as a form of adversarial noise, it could negatively impact the performance of deep learning-based methods, which are now the de-facto standard for dense stereo vision. In this paper, we propose the Active-Passive SimS…
Zhejiang University（浙江大学）；HwaMei Hospital, University of Chinese Academy of Sciences（中国科学院大学华美医院）；Ningbo Institute of Life and Health Industry, University of Chinese Academy of Sciences（中国科学院大学宁波生命与健康产业研究院）
Electroencephalography (EEG) based emotion recognition has demonstrated tremendous improvement in recent years. Specifically, numerous domain adaptation (DA) algorithms have been exploited in the past five years to enhance the generalization of emotion recognition models across subjects. The DA methods assume that calibration data (although unlabeled) exists in the target domain (new user). However, this assumption conflicts with the application scenario that the model should be deployed without the time-consuming calibration experiments. We argue that domain generalization (DG) is more reason…
Benchmarking Test-Time Adaptation against Distribution Shifts in Image Classification
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；University of Science and Technology of China（中国科学技术大学）
BenchHAR: Benchmarking Self-Supervised Learning for Generalizable Sensor-based Activity Recognition
The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Peking University（北京大学）；Singapore-MIT Alliance for Research and Technology（新加坡-麻省理工联合研究技术联盟）
Benchmarking Retinal Blood Vessel Segmentation Models for Cross-Dataset and Cross-Disease Generalization
University of Ibadan（伊巴丹大学）；African Institute for Mathematical Sciences（非洲数学科学研究所）；Hertie Institute for AI in Brain Health（赫蒂脑健康人工智能研究所）；University of Tübingen（蒂宾根大学）；University of Bern（伯尔尼大学）；University College London（伦敦大学学院）；Tübingen AI Center（蒂宾根人工智能中心）
VLM4Cluster: Benchmarking Deep Clustering In the Era of Vision-Language Pre-training
University College London（伦敦大学学院）；The University of Queensland（昆士兰大学）；Southeast University（东南大学）；University of Science and Technology of China（中国科学技术大学）；Auckland University of Technology（奥克兰理工大学）
UniIR: Training and Benchmarking Universal Multimodal Information Retrievers
University of Waterloo（滑铁卢大学）；Georgia Institute of Technology（佐治亚理工学院）；Hong Kong University of Science and Technology（香港科技大学）；Google DeepMind（谷歌DeepMind）
Morphing attacks have posed a severe threat to Face Recognition System (FRS). Despite the number of advancements reported in recent works, we note serious open issues such as independent benchmarking, generalizability challenges and considerations to age, gender, ethnicity that are inadequately addressed. Morphing Attack Detection (MAD) algorithms often are prone to generalization challenges as they are database dependent. The existing databases, mostly of semi-public nature, lack in diversity in terms of ethnicity, various morphing process and post-processing pipelines. Further, they do not r…
Rapid-Rich Object Search (ROSE) Lab and the College of Computing and Data Science, Nanyang Technological University (NTU)（快速丰富对象搜索（ROSE）实验室和南洋理工大学计算与数据科学学院）；School of Computing and Information Technology and Dongguan Key Laboratory for Intelligence and Information Technology, Great Bay University（计算与信息科技学院和东莞智能与信息技术重点实验室，大湾大学）；DSO National Laboratories（国防科学实验室）；College of Computing and Data Science, Nanyang Technological University (NTU)（计算与数据科学学院，南洋理工大学）；SMBU, Shenzhen 518172, China（深圳SMBU，越南河内VinUniversity，和新加坡NTU）；VinUniversity, Hanoi 100000, Vietnam；and NTU, Singapore
Nanyang Technological University（南洋理工大学）；City University of Hong Kong（香港城市大学）
The state-of-the-art deep neural networks are vulnerable to common corruptions (e.g., input data degradations, distortions, and disturbances caused by weather changes, system error, and processing). While much progress has been made in analyzing and improving the robustness of models in image understanding, the robustness in video understanding is largely unexplored. In this paper, we establish a corruption robustness benchmark, Mini Kinetics-C and Mini SSV2-C, which considers temporal corruptions beyond spatial corruptions in images. We make the first attempt to conduct an exhaustive study on…
Polyp-E: Benchmarking the Robustness of Deep Segmentation Models via Polyp Editing
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；The Fifth Medical Center, Chinese PLA General Hospital（中国人民解放军总医院第五医学中心）；Children’s Hospital of Zhejiang University School of Medicine（浙江大学医学院附属儿童医院）
Benchmarking Ophthalmology Foundation Models for Clinically Significant Age Macular Degeneration Detection
Faculty of Biomedical Engineering, Technion-IIT（技术学院生物医学工程系）；Taub Faculty of Computer Science, Technion-IIT（技术学院计算机科学系）；Faculty of Mathematics, Technion-IIT（技术学院数学系）；Department of Ophthalmology, Hillel Yaffe Medical Center（希勒尔·亚法医疗中心眼科部）；The Ruth and Rappaport Faculty of Medicine, Technion-IIT（技术学院医学院）；Ophthalmology Department, São Paulo Federal University（圣保罗联邦大学眼科部）；Medical Engineering and Science, Massachusetts Institute of Technology（麻省理工学院医学工程与科学系）；The Adelson School of Medicine, Ariel University（阿里尔大学阿德勒医学学校）
SemSegBench & DetecBench: Benchmarking Reliability and Generalization Beyond Classification
Data and Web Science Group, University of Mannheim（曼海姆大学数据与网络科学小组）；Max-Planck-Institute for Informatics, Saarland Informatics Campus（马克斯·普朗克信息研究所，萨尔兰信息校园）
Benchmarking Interaction, Beyond Policy: a Reproducible Benchmark for Collaborative Instance Object Navigation
Sapienza University of Rome（罗马大学）；University of Verona（维罗纳大学）；Polytechnic of Turin（都灵理工大学）；Fondazione Bruno Kessler（布鲁诺·凯斯勒基金会）；Reykjavik University（雷克雅未克大学）
We present a machine sound dataset to benchmark domain generalization techniques for anomalous sound detection (ASD). Domain shifts are differences in data distributions that can degrade the detection performance, and handling them is a major issue for the application of ASD systems. While currently available datasets for ASD tasks assume that occurrences of domain shifts are known, in practice, they can be difficult to detect. To handle such domain shifts, domain generalization techniques that perform well regardless of the domains should be investigated. In this paper, we present the first A…
We present a simple domain generalization baseline, which wins second place in both the common context generalization track and the hybrid context generalization track respectively in NICO CHALLENGE 2022. We verify the founding in recent literature, domainbed, that ERM is a strong baseline compared to recent state-of-the-art domain generalization methods and propose SimpleDG which includes several simple yet effective designs that further boost generalization performance. Code is available at https://github.com/megvii-research/SimpleDG
Approaches based on deep neural networks have achieved striking performance when testing data and training data share similar distribution, but can significantly fail otherwise. Therefore, eliminating the impact of distribution shifts between training and testing data is crucial for building performance-promising deep models. Conventional methods assume either the known heterogeneity of training data (e.g. domain labels) or the approximately equal capacities of different domains. In this paper, we consider a more challenging case where neither of the above assumptions holds. We propose to addr…
