Simple Control Baselines for Evaluating Transfer Learning
Swiss Federal Institute of Technology (EPFL)（瑞士联邦理工学院（EPFL））；Moscow Institute of Physics Technology (MIPT)（莫斯科物理技术学院（MIPT））；transfer-controls.epfl.ch（transfer-controls.epfl.ch（网址，非机构））
Andrei Atanov, Shijian Xu, Onur Beker, Andrei Filatov, Amir Zamir
Transfer learning has witnessed remarkable progress in recent years, for example, with the introduction of augmentation-based contrastive self-supervised learning methods. While a number of large-scale empirical studies on the transfer performance of such models have been conducted, there is not yet an agreed-upon set of control baselines, evaluation practices, and metrics to report, which often hinders a nuanced and calibrated understanding of the real efficacy of the methods. We share an evaluation standard that aims to quantify and communicate transfer learning performance in an informative and accessible setup. This is done by baking a number of simple yet critical control baselines in the evaluation method, particularly the blind-guess (quantifying the dataset bias), scratch-model (quantifying the architectural contribution), and maximal-supervision (quantifying the upper-bound). To demonstrate how the evaluation standard can be employed, we provide an example empirical study investigating a few basic questions about self-supervised learning. For example, using this standard, the study shows the effectiveness of existing self-supervised pre-training methods is skewed towards image classification tasks versus dense pixel-wise predictions. In general, we encourage using/reporting the suggested control baselines in evaluating transfer learning in order to gain a more meaningful and informative understanding.
National Technical University of Athens（雅典国立技术大学）；Institute of Communications and Computer Systems（通信与计算机系统研究所）
Recent studies indicate that detecting radiographic patterns on CT scans can yield high sensitivity and specificity for Covid-19 localization. In this paper, we investigate the appropriateness of deep learning models transferability, for semantic segmentation of pneumonia-infected areas in CT images. Transfer learning allows for the fast initialization/reutilization of detection models, given that large volumes of training data are not available. Our work explores the efficacy of using pre-trained U-Net architectures, on a specific CT data set, for identifying Covid-19 side-effects over images…
Continual learning has received a great deal of attention recently with several approaches being proposed. However, evaluations involve a diverse set of scenarios making meaningful comparison difficult. This work provides a systematic categorization of the scenarios and evaluates them within a consistent framework including strong baselines and state-of-the-art methods. The results provide an understanding of the relative difficulty of the scenarios and that simple baselines (Adagrad, L2 regularization, and naive rehearsal strategies) can surprisingly achieve similar performance to current mai…
X-ray imagery security screening is essential to maintaining transport security against a varying profile of threat or prohibited items. Particular interest lies in the automatic detection and classification of weapons such as firearms and knives within complex and cluttered X-ray security imagery. Here, we address this problem by exploring various end-to-end object detection Convolutional Neural Network (CNN) architectures. We evaluate several leading variants spanning the Faster R-CNN, Mask R-CNN, and RetinaNet architectures to explore the transferability of such models between varying X-ray…
Stony Brook Medicine（石溪大学医学中心）；Stony Brook University（石溪大学）；Cold Spring Harbor Laboratory（冷泉港实验室）
Histopathology remains the gold standard for diagnosis of various cancers. Recent advances in computer vision, specifically deep learning, have facilitated the analysis of histopathology images for various tasks, including immune cell detection and microsatellite instability classification. The state-of-the-art for each task often employs base architectures that have been pretrained for image classification on ImageNet. The standard approach to develop classifiers in histopathology tends to focus narrowly on optimizing models for a single task, not considering the aspects of modeling innovatio…
Evaluating transfer learning strategies for improving dairy cattle body weight prediction in small farms using depth-image and point-cloud data
Department of Animal Sciences, University of Florida（动物科学系，佛罗里达大学）；Department of Large Animal Clinical Sciences, University of Florida（大动物临床科学系，佛罗里达大学）；Laboratory of Biometry and Bioinformatics, Department of Agricultural and Environmental Biology, Graduate School of Agricultural and Life Sciences, The University of Tokyo（生物度量与生物信息学实验室，农业与环境生物学系，东京大学研究生院）；Department of Agricultural and Environmental Biology, Graduate School of Agricultural and Life Sciences, The University of Tokyo（农业与环境生物学系，东京大学研究生院）
Yonsei University（延世大学）；University of Tuebingen（图宾根大学）
Weakly-supervised object localization (WSOL) has gained popularity over the last years for its promise to train localization models with only image-level labels. Since the seminal WSOL work of class activation mapping (CAM), the field has focused on how to expand the attention regions to cover objects more broadly and localize them better. However, these strategies rely on full localization supervision to validate hyperparameters and for model selection, which is in principle prohibited under the WSOL setup. In this paper, we argue that WSOL task is ill-posed with only image-level labels, and …
Edith Cowan University（伊迪丝·考恩大学）；University of Western Australia（西澳大利亚大学）
Deep learning is gaining instant popularity in computer aided diagnosis of COVID-19. Due to the high sensitivity of Computed Tomography (CT) to this disease, CT-based COVID-19 detection with visual models is currently at the forefront of medical imaging research. Outcomes published in this direction are frequently claiming highly accurate detection under deep transfer learning. This is leading medical technologists to believe that deep transfer learning is the mainstream solution for the problem. However, our critical analysis of the literature reveals an alarming performance disparity between…
Protein modeling is an increasingly popular area of machine learning research. Semi-supervised learning has emerged as an important paradigm in protein modeling due to the high cost of acquiring supervised protein labels, but the current literature is fragmented when it comes to datasets and standardized evaluation techniques. To facilitate progress in this field, we introduce the Tasks Assessing Protein Embeddings (TAPE), a set of five biologically relevant semi-supervised learning tasks spread across different domains of protein biology. We curate tasks into specific training, validation, an…
Convolutional Neural Networks (CNNs) have revolutionized performances in several machine learning tasks such as image classification, object tracking, and keyword spotting. However, given that they contain a large number of parameters, their direct applicability into low resource tasks is not straightforward. In this work, we experiment with an application of CNN models to gastrointestinal landmark classification with only a few thousands of training samples through transfer learning. As in a standard transfer learning approach, we train CNNs on a large external corpus, followed by representat…
Technische Universität Berlin（柏林工业大学）；Max Planck Institute for Human Development（马克斯·普朗克人类发展研究所）；Stanford University（斯坦福大学）；Max Planck UCL Centre for Computational Psychiatry and Ageing Research（马克斯·普朗克伦敦大学学院计算精神病学与老龄化研究中心）；Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希·赫兹研究所）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所）；Korea University（高丽大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
Research in many fields has shown that transfer learning (TL) is well-suited to improve the performance of deep learning (DL) models in datasets with small numbers of samples. This empirical success has triggered interest in the application of TL to cognitive decoding analyses with functional neuroimaging data. Here, we systematically evaluate TL for the application of DL models to the decoding of cognitive states (e.g., viewing images of faces or houses) from whole-brain functional Magnetic Resonance Imaging (fMRI) data. We first pre-train two DL architectures on a large, public fMRI dataset …
Input perturbation methods occlude parts of an input to a function and measure the change in the function's output. Recently, input perturbation methods have been applied to generate and evaluate saliency maps from convolutional neural networks. In practice, neutral baseline images are used for the occlusion, such that the baseline image's impact on the classification probability is minimal. However, in this paper we show that arguably neutral baseline images still impact the generated saliency maps and their evaluation with input perturbations. We also demonstrate that many choices of hyperpa…
Towards Good Practices in Evaluating Transfer Adversarial Attacks
CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）；LIS - Ecole Centrale Marseille（马赛中央理工学院信息科学实验室）；SKLCS, Institute of Software, CAS（中国科学院软件研究所计算机科学国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Inria, Univ Rennes, CNRS, IRISA（法国国家信息与自动化研究所，雷恩大学，法国国家科学研究中心，IRISA）
Public Health Ontario（安大略省公共卫生局）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）
Deep learning and knowledge transfer techniques have permeated the field of medical imaging and are considered as key approaches for revolutionizing diagnostic imaging practices. However, there are still challenges for the successful integration of deep learning into medical imaging tasks due to a lack of large annotated imaging data. To address this issue, we propose a teacher-student learning framework to transfer knowledge from a carefully pre-trained convolutional neural network (CNN) teacher to a student CNN. In this study, we explore the performance of knowledge transfer in the medical i…
Tsinghua University（清华大学）；Microsoft Research Asia（微软亚洲研究院）
This paper proposes a simple transfer learning baseline for sign language translation. Existing sign language datasets (e.g. PHOENIX-2014T, CSL-Daily) contain only about 10K-20K pairs of sign videos, gloss annotations and texts, which are an order of magnitude smaller than typical parallel data for training spoken language translation models. Data is thus a bottleneck for training effective sign language translation models. To mitigate this problem, we propose to progressively pretrain the model from general-domain datasets that include a large amount of external supervision to within-domain d…
Microsoft Research Asia（微软亚洲研究院）；University of Electronic Science and Technology of China（电子科技大学）
There has been significant progress on pose estimation and increasing interests on pose tracking in recent years. At the same time, the overall algorithm and system complexity increases as well, making the algorithm analysis and comparison more difficult. This work provides simple and effective baseline methods. They are helpful for inspiring and evaluating new ideas for the field. State-of-the-art results are achieved on challenging benchmarks. The code will be available at https://github.com/leoxiaobin/pose.pytorch.
Although CNNs have gained the ability to transfer learned knowledge from source task to target task by virtue of large annotated datasets but consume huge processing time to fine-tune without GPU. In this paper, we propose a new computationally efficient transfer learning approach using classification layer features of pre-trained CNNs by appending layer after existing classification layer. We demonstrate that fine-tuning of the appended layer with existing classification layer for new task converges much faster than baseline and in average outperforms baseline classification accuracy. Further…
