Machine learning (ML) models are only as good as the data they are trained on. But recent studies have found datasets widely used to train and evaluate ML models, e.g. ImageNet, to have pervasive labeling errors. Erroneous labels on the train set hurt ML models' ability to generalize, and they impact evaluation and model selection using the test set. Consequently, learning in the presence of labeling errors is an active area of research, yet this field lacks a comprehensive benchmark to evaluate these methods. Most of these methods are evaluated on a few computer vision datasets with significant variance in the experimental protocols. With such a large pool of methods and inconsistent evaluation, it is also unclear how ML practitioners can choose the right models to assess label quality in their data. To this end, we propose a benchmarking environment AQuA to rigorously evaluate methods that enable machine learning in the presence of label noise. We also introduce a design space to delineate concrete design choices of label error detection models. We hope that our proposed design space and benchmark enable practitioners to choose the right tools to improve their label quality and that our benchmark enables objective and rigorous evaluation of machine learning tools facing mislabeled data.
Audio-Visual Class-Incremental Learning for Fish Feeding intensity Assessment in Aquaculture
Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey(视觉、语音和信号处理中心(CVSSP), Surrey大学);College of Intelligence and Computing, Tianjin University(智能与计算学院,天津大学);School of Computer and Communication Engineering, University of Science and Technology Beijing(计算机与通信工程学院,北京科技大学);National Innovation Center for Digital Fishery, China Agricultural University(数字渔业国家创新中心,中国农业大学)
AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answering
Graduate Institute of Communication Engineering, National Taiwan University, Taiwan(台湾国立台湾大学通信工程研究所);Artificial Intelligence Center of Research Excellence (AI-CoRE), National Taiwan University, Taiwan(台湾国立台湾大学人工智能研究中心)
AMPLIFAI: A Multiphase CT Dataset for Benchmarking Clinical Reasoning in LI-RADS Assessment of Liver Lesions
University of Maryland, College Park(马里兰大学帕克分校);University of Maryland School of Medicine(马里兰大学医学院);University of Maryland Institute for Health Computing(马里兰大学健康计算研究所);University of Maryland Medical System(马里兰大学医学系统)
MOS-Bench: Benchmarking Generalization Abilities of Subjective Speech Quality Assessment Models
Graduate School of Informatics, Nagoya University, Japan(名古屋大学信息科学研究生院,日本);National Institute of Information and Communications Technology (NICT), Japan(日本信息通信技术研究所);Information Technology Center, Nagoya University, Japan(名古屋大学信息技术中心,日本)
CSAW-M: An Ordinal Classification Dataset for Benchmarking Mammographic Masking of Cancer
KTH Royal Institute of Technology(皇家理工学院);SciLifeLab;Karolinska Institutet(卡罗林斯卡研究所);Karolinska University Hospital(卡罗林斯卡大学医院);Saint Göran Hospital(圣戈兰医院)
Benchmarking Deep Learning-Based Reconstruction Methods for Photoacoustic Computed Tomography with Clinically Relevant Synthetic Datasets
Department of Bioengineering, University of Illinois Urbana-Champaign(生物工程系,伊利诺伊大学厄巴纳-香槟分校);Department of Electrical and Computer Engineering, University of Illinois Urbana-Champaign(电气与计算机工程系,伊利诺伊大学厄巴纳-香槟分校);Department of Biomedical Engineering, The University of Texas at Austin(生物医学工程系,德克萨斯大学奥斯汀分校);Oden Institute for Computational Engineering and Sciences, The University of Texas at Austin(奥登计算工程与科学研究所,德克萨斯大学奥斯汀分校)
Benchmarking and Enhancing PPG-Based Cuffless Blood Pressure Estimation Methods
Department of Engineering Technology, University of Houston, Texas, USA(工程科技系,休斯顿大学,德克萨斯州,美国);Department of Electrical and Computer Engineering, University of Houston, Texas, USA(电气与计算机工程系,休斯顿大学,德克萨斯州,美国);Department of Information Science Technology, University of Houston, Texas, USA(信息科学科技系,休斯顿大学,德克萨斯州,美国);Department of Cardiovascular Surgery, Houston Methodist Hospital, Texas, USA(心血管外科系,休斯顿方法医院,德克萨斯州,美国);Department of Biomedical Engineering, University of Houston, Texas, USA(生物医学工程系,休斯顿大学,德克萨斯州,美国)
Benchmarking community drug response prediction models: datasets, models, tools, and metrics for cross-dataset generalization analysis
Argonne National Laboratory(阿贡国家实验室);Frederick National Laboratory for Cancer Research(弗雷德里克国家癌症实验室);University of Nebraska–Lincoln(内布拉斯加大学林肯分校);Texas Tech University(德克萨斯理工大学);Los Alamos National Laboratory(洛斯阿拉莫斯国家实验室)
SciHorizon: Benchmarking AI-for-Science Readiness from Scientific Data to Large Language Models
Computer Network Information Center, Chinese Academy of Sciences(中国科学院计算机网络信息中心);University of Chinese Academy of Sciences(中国科学院大学);University of Science and Technology of China(中国科学技术大学);Hefei University of Technology(合肥工业大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州))
A robust and scalable framework for hallucination detection in virtual tissue staining and digital pathology
University of California, Los Angeles(加利福尼亚大学洛杉矶分校);California NanoSystems Institute (CNSI)(加州纳米系统研究所);Hadassah Hebrew University Medical Center(哈达萨希伯来大学医学中心);University of Southern California(南加利福尼亚大学)