Comparative Benchmarking of Failure Detection Methods in Medical Image Segmentation: Unveiling the Role of Confidence Aggregation
German Cancer Research Center (DKFZ)（德国癌症研究中心）；Medical Faculty Heidelberg, Heidelberg University（海德堡大学海德堡医学院）；Helmholtz Imaging（亥姆霍兹成像）；Heidelberg University Hospital（海德堡大学医院）
Maximilian Zenk, David Zimmerer, Fabian Isensee, Jeremias Traub, Tobias Norajitra, Paul F. Jäger, Klaus Maier-Hein
备注：
This work has been submitted for possible publication
Semantic segmentation is an essential component of medical image analysis research, with recent deep learning algorithms offering out-of-the-box applicability across diverse datasets. Despite these advancements, segmentation failures remain a significant concern for real-world clinical applications, necessitating reliable detection mechanisms. This paper introduces a comprehensive benchmarking framework aimed at evaluating failure detection methodologies within medical image segmentation. Through our analysis, we identify the strengths and limitations of current failure detection metrics, advocating for the risk-coverage analysis as a holistic evaluation approach. Utilizing a collective dataset comprising five public 3D medical image collections, we assess the efficacy of various failure detection strategies under realistic test-time distribution shifts. Our findings highlight the importance of pixel confidence aggregation and we observe superior performance of the pairwise Dice score (Roy et al., 2019) between ensemble predictions, positioning it as a simple and robust baseline for failure detection in medical image segmentation. To promote ongoing research, we make the benchmarking framework available to the community.
University of Alberta（阿尔伯塔大学）；Valeo Vision Systems（法雷奥视觉系统公司）；Cairo University（开罗大学）
Semantic segmentation benefits robotics related applications especially autonomous driving. Most of the research on semantic segmentation is only on increasing the accuracy of segmentation models with little attention to computationally efficient solutions. The few work conducted in this direction does not provide principled methods to evaluate the different design choices for segmentation. In this paper, we address this gap by presenting a real-time semantic segmentation benchmarking framework with a decoupled design for feature extraction and decoding methods. The framework is comprised of d…
Few-Shot Object Detection (FSOD) methods are mainly designed and evaluated on natural image datasets such as Pascal VOC and MS COCO. However, it is not clear whether the best methods for natural images are also the best for aerial images. Furthermore, direct comparison of performance between FSOD methods is difficult due to the wide variety of detection frameworks and training strategies. Therefore, we propose a benchmarking framework that provides a flexible environment to implement and compare attention-based FSOD methods. The proposed framework focuses on attention mechanisms and is divided…
Vision Transformers for Zero-Shot Clustering of Animal Images: A Comparative Benchmarking Study
Department of Chemistry and Bioscience（化学与生物科学系）；Aalborg University（奥尔堡大学）；Department of Architecture, Design and Media Technology（建筑、设计与媒体技术系）；Visual Analysis and Perception Lab（视觉分析与感知实验室）
Institute for Systems and Robotics（系统与机器人研究所）；Instituto Superior Técnico（高等技术学院）；Champalimaud Centre for the Unknown（尚帕利莫未知中心）
A defining characteristic of intelligent systems is the ability to make action decisions based on the anticipated outcomes. Video prediction systems have been demonstrated as a solution for predicting how the future will unfold visually, and thus, many models have been proposed that are capable of predicting future frames based on a history of observed frames~(and sometimes robot actions). However, a comprehensive method for determining the fitness of different video prediction models at guiding the selection of actions is yet to be developed. Current metrics assess video prediction models bas…
Over the past decades, various super-resolution (SR) techniques have been developed to enhance the spatial resolution of digital images. Despite the great number of methodical contributions, there is still a lack of comparative validations of SR under practical conditions, as capturing real ground truth data is a challenging task. Therefore, current studies are either evaluated 1) on simulated data or 2) on real data without a pixel-wise ground truth. To facilitate comprehensive studies, this paper introduces the publicly available Super-Resolution Erlangen (SupER) database that includes real …
VisioMath: Benchmarking Figure-based Mathematical Reasoning in LMMs
School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；Beijing Key Laboratory of Artificial Intelligence for Education（北京人工智能教育重点实验室）；Engineering Research Center of Intelligent Technology and Educational Application, Ministry of Education（教育部智能技术与教育应用工程研究中心）
Univ. Rennes（雷恩大学）；INSA Rennes（雷恩国立应用科学学院）；IETR - UMR CNRS 6164（法国国家科学研究中心6164联合研究单元——雷恩电子与电信研究所）；DGA-MI（法国国防部军备总局-信息与系统局）
Image denoising has recently taken a leap forward due to machine learning. However, image denoisers, both expert-based and learning-based, are mostly tested on well-behaved generated noises (usually Gaussian) rather than on real-life noises, making performance comparisons difficult in real-world conditions. This is especially true for learning-based denoisers which performance depends on training data. Hence, choosing which method to use for a specific denoising problem is difficult. This paper proposes a comparative study of existing denoisers, as well as an extensible open tool that makes it…
CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Dataset
School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；First Affiliated Hospital of Anhui University of Chinese Medicine（安徽中医药大学第一附属医院）
Out of distribution (OOD) detection is a crucial part of making machine learning systems robust. The ImageNet-O dataset is an important tool in testing the robustness of ImageNet trained deep neural networks that are widely used across a variety of systems and applications. We aim to perform a comparative analysis of OOD detection methods on ImageNet-O, a first of its kind dataset with a label distribution different than that of ImageNet, that has been created to aid research in OOD detection for ImageNet models. As this dataset is fairly new, we aim to provide a comprehensive benchmarking of …
