The shift towards end-to-end deep learning has brought unprecedented advances in many areas of computer vision. However, deep neural networks are trained on images with resolutions that rarely exceed $1,000 \times 1,000$ pixels. The growing use of scanners that create images with extremely high resolutions (average can be $100,000 \times 100,000$ pixels) thereby presents novel challenges to the field. Most of the published methods preprocess high-resolution images into a set of smaller patches, imposing an a priori belief on the best properties of the extracted patches (magnification, field of view, location, etc.). Herein, we introduce Magnifying Networks (MagNets) as an alternative deep learning solution for gigapixel image analysis that does not rely on a preprocessing stage nor requires the processing of billions of pixels. MagNets can learn to dynamically retrieve any part of a gigapixel image, at any magnification level and field of view, in an end-to-end fashion with minimal ground truth (a single global, slide-level label). Our results on the publicly available Camelyon16 and Camelyon17 datasets corroborate to the effectiveness and efficiency of MagNets and the proposed optimization framework for whole slide image classification. Importantly, MagNets process far less patches from each slide than any of the existing approaches ($10$ to $300$ times less).
Image zooming is the process of enlarging the spatial resolution of a given digital image. We present a novel technique that intelligently modifies the classical pixel replication method for zooming. Our method decomposes a given image into layer of binary images, interpolates them by magnifying the binary patterns preserving their geometric shape and finally aggregates them all to obtain the zoomed image. Although the quality of our zoomed images is much higher than that of nearest neighbor and bilinear interpolation and comparable with bicubic interpolation, the running time of our technique…
The pursuit of smaller pixel sizes at ever increasing resolution in digital image sensors is mainly driven by the stringent price and form-factor requirements of sensors and optics in the cellular phone market. Recently, Eric Fossum proposed a novel concept of an image sensor with dense sub-diffraction limit one-bit pixels jots, which can be considered a digital emulation of silver halide photographic film. This idea has been recently embodied as the EPFL Gigavision camera. A major bottleneck in the design of such sensors is the image reconstruction process, producing a continuous high dynamic…
The first global agricultural field boundary map at 10m resolution
Microsoft AI for Good Research Lab（微软AI for Good研究实验室）；Arizona State University（亚利桑那州立大学）；Washington University in St. Louis（圣路易斯华盛顿大学）；Oregon State University（俄勒冈州立大学）；Clark University（克拉克大学）；Taylor Geospatial（泰勒地理空间）
We present a system for performing visual search over billions of aerial and satellite images. The purpose of visual search is to find images that are visually similar to a query image. We define visual similarity using 512 abstract visual features generated by a convolutional neural network that has been trained on aerial and satellite imagery. The features are converted to binary values to reduce data and compute requirements. We employ a hash-based search using Bigtable, a scalable database service from Google Cloud. Searching the continental United States at 1-meter pixel resolution, corre…
Multiple instance learning (MIL) is a key algorithm for classification of whole slide images (WSI). Histology WSIs can have billions of pixels, which create enormous computational and annotation challenges. Typically, such images are divided into a set of patches (a bag of instances), where only bag-level class labels are provided. Deep learning based MIL methods calculate instance features using convolutional neural network (CNN). Our proposed approach is also deep learning based, with the following two contributions: Firstly, we propose to explicitly account for dependencies between instance…
Most state-of-the-art techniques for multi-class image segmentation and labeling use conditional random fields defined over pixels or image regions. While region-level models often feature dense pairwise connectivity, pixel-level models are considerably larger and have only permitted sparse graph structures. In this paper, we consider fully connected CRF models defined on the complete set of pixels in an image. The resulting graphs have billions of edges, making traditional inference algorithms impractical. Our main contribution is a highly efficient approximate inference algorithm for fully c…
University of Wisconsin-Milwaukee（威斯康星大学密尔沃基分校）；Institute for Pathophysiology and Allergy Research, Medical University of Vienna（维也纳医科大学病理生理学与过敏研究所）；Bournemouth University（伯恩茅斯大学）；Advancing the Zenith of Healthcare (AZH) Wound and Vascular Center（AZH伤口与血管中心）
Acute and chronic wounds with varying etiologies burden the healthcare systems economically. The advanced wound care market is estimated to reach $22 billion by 2024. Wound care professionals provide proper diagnosis and treatment with heavy reliance on images and image documentation. Segmentation of wound boundaries in images is a key component of the care and diagnosis protocol since it is important to estimate the area of the wound and provide quantitative measurement for the treatment. Unfortunately, this process is very time-consuming and requires a high level of expertise. Recently autom…
Mask Clustering-based Annotation Engine for Large-Scale Submeter Land Cover Mapping
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing (LIESMARS)（信息工程测绘遥感国家重点实验室）；Wuhan University（武汉大学）；School of Artificial Intelligence（人工智能学院）；Chinese Antarctic Center of Surveying and Mapping（测绘南极中心）
The widespread adoption of whole slide imaging has increased the demand for effective and efficient gigapixel image analysis. Deep learning is at the forefront of computer vision, showcasing significant improvements over previous methodologies on visual understanding. However, whole slide images have billions of pixels and suffer from high morphological heterogeneity as well as from different types of artefacts. Collectively, these impede the conventional use of deep learning. For the clinical translation of deep learning solutions to become a reality, these challenges need to be addressed. In…
Exploring shared memory architectures for end-to-end gigapixel deep learning
Vanderbilt University（范德堡大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；National Institutes of Health（美国国立卫生研究院）；Vanderbilt University Medical Center（范德堡大学医学中心）；Veterans Affairs Tennessee Valley Healthcare System（退伍军人事务部田纳西谷医疗系统）
Google Brain（谷歌大脑）；Stony Brook University（石溪大学）；Mayo Clinic（妙佑医疗国际）
Medical images such as 3D computerized tomography (CT) scans and pathology images, have hundreds of millions or billions of voxels/pixels. It is infeasible to train CNN models directly on such high resolution images, because neural activations of a single image do not fit in the memory of a single GPU/TPU, and naive data and model parallelism approaches do not work. Existing image analysis approaches alleviate this problem by cropping or down-sampling input images, which leads to complicated implementation and sub-optimal performance due to information loss. In this paper, we implement spatial…
Learning with less: label-efficient land cover classification at very high spatial resolution using self-supervised deep learning
Department of Agricultural and Biological Engineering, Mississippi State University（农业与生物工程系，密苏里州立大学）；North Mississippi Research and Extension Center, Mississippi State University（北密苏里研究与推广中心，密苏里州立大学）
Uncertainty-Aware Hourly Air Temperature Mapping at 2 km Resolution via Physics-Guided Deep Learning
Spatial Sciences Institute（空间科学研究所）；Dornsife College of Letters, Arts and Sciences（文理学院）；University of Southern California（南加州大学）；Division of Biostatistics, Department of Population and Public Health Sciences（流行病学与公共卫生科学系生物统计学部）；Keck School of Medicine（凯克医学院）
Centre for Medical Image Computing, University College London（伦敦大学学院医学图像计算中心）；Healthcare Intelligence, Microsoft Research Cambridge（微软剑桥研究院医疗智能）
Segmentation of ultra-high resolution images is challenging because of their enormous size, consisting of millions or even billions of pixels. Typical solutions include dividing input images into patches of fixed size and/or down-sampling to meet memory constraints. Such operations incur information loss in the field-of-view (FoV) i.e., spatial coverage and the image resolution. The impact on segmentation performance is, however, as yet understudied. In this work, we start with a motivational experiment which demonstrates that the trade-off between FoV and resolution affects the segmentation p…
The Gleason grading system using histological images is the most powerful diagnostic and prognostic predictor of prostate cancer. The current standard inspection is evaluating Gleason H&E-stained histopathology images by pathologists. However, it is complicated, time-consuming, and subject to observers. Deep learning (DL) based-methods that automatically learn image features and achieve higher generalization ability have attracted significant attention. However, challenges remain especially using DL to train the whole slide image (WSI), a predominant clinical source in the current diagnostic s…
ONERA（法国国家航空航天研究院）；Université Paris-Saclay（巴黎萨克雷大学）；Université Bretagne Sud（南布列塔尼大学）；IRISA（布列塔尼国立高等电信学校计算机科学研究所）；European Space Agency（欧洲空间局）；Conservatoire national des arts et métiers（法国国立工艺学院）；CEDRIC（计算机科学、电子学与自动化研究中心）
The development of semi-supervised learning techniques is essential to enhance the generalization capacities of machine learning algorithms. Indeed, raw image data are abundant while labels are scarce, therefore it is crucial to leverage unlabeled inputs to build better models. The availability of large databases have been key for the development of learning algorithms with high level performance. Despite the major role of machine learning in Earth Observation to derive products such as land cover maps, datasets in the field are still limited, either because of modest surface coverage, lack of…
Texas A&M University（德克萨斯农工大学）；Mississippi State University（密西西比州立大学）；U.S.D.A. Agriculture Research Service（美国农业部农业研究局）；U.S.D.A.- APHIS PPQ S&T（美国农业部动植物卫生检验局植物保护与检疫科学技术处）；Utah State University（犹他州立大学）；Texas A&M AgriLife Research & Extension Center（德克萨斯农工大学生命科学研究与推广中心）
The cotton boll weevil, Anthonomus grandis Boheman is a serious pest to the U.S. cotton industry that has cost more than 16 billion USD in damages since it entered the United States from Mexico in the late 1800s. This pest has been nearly eradicated; however, southern part of Texas still faces this issue and is always prone to the pest reinfestation each year due to its sub-tropical climate where cotton plants can grow year-round. Volunteer cotton (VC) plants growing in the fields of inter-seasonal crops, like corn, can serve as hosts to these pests once they reach pin-head square stage (5-6 l…
