Virtual microscopy includes digitisation of histology slides and the use of computer technologies for complex investigation of diseases such as cancer. However, automated image analysis, or website publishing of such digital images, is hampered by their large file sizes. We have developed two Java based open source tools: Snapshot Creator and NDPI-Splitter. Snapshot Creator converts a portion of a large digital slide into a desired quality JPEG image. The image is linked to the patients clinical and treatment information in a customised open source cancer data management software (Caisis) in use at the Australian Breast Cancer Tissue Bank (ABCTB) and then published on the ABCTB website www.abctb.org.au using Deep Zoom open source technology. Using the ABCTB online search engine, digital images can be searched by defining various criteria such as cancer type, or biomarkers expressed. NDPI-Splitter splits a large image file into smaller sections of TIFF images so that they can be easily analysed by image analysis software such as Metamorph or Matlab. NDPI-Splitter also has the capacity to filter out empty images. Snapshot Creator and NDPI-Splitter are novel open source Java tools. They convert digital slides into files of smaller size for further processing. In conjunction with other open source tools such as Deep Zoom and Caisis, this suite of tools is used for the management and archiving of digital microscopy images, enabling digitised images to be explored and zoomed online. Our online image repository also has the capacity to be used as a teaching resource. These tools also enable large files to be sectioned for image analysis.
Stony Brook University（石溪大学）；University of Brasília（巴西利亚大学）；Emory University（埃默里大学）
An accurate steatosis quantification with pathology tissue samples is of high clinical importance. However, such pathology measurement is manually made in most clinical practices, subject to severe reader variability due to large sampling bias and poor reproducibility. Although some computerized automated methods are developed to quantify the steatosis regions, they present limited analysis capacity for high resolution whole-slide microscopy images and accurate overlapped steatosis division. In this paper, we propose a method that extracts an individual whole tissue piece at high resolution wi…
University College London（伦敦大学学院）；University of Ibadan（伊巴丹大学）；University College Hospital, Ibadan（伊巴丹大学学院医院）
Fast accurate diagnosis of malaria is still a global health challenge for which automated digital-pathology approaches could provide scalable solutions amenable to be deployed in low-to-middle income countries. Here we address the problem of Extended Depth-of-Field (EDoF) in thick blood film microscopy for rapid automated malaria diagnosis. High magnification oil-objectives (100x) with large numerical aperture are usually preferred to resolve the fine structural details that help separate true parasites from distractors. However, such objectives have a very limited depth-of-field requiring the…
Histopathology images; microscopy images of stained tissue biopsies contain fundamental prognostic information that forms the foundation of pathological analysis and diagnostic medicine. However, diagnostics from histopathology images generally rely on a visual cognitive assessment of tissue slides which implies an inherent element of interpretation and hence subjectivity. Access to digitized histopathology images enabled the development of computational systems aiming at reducing manual intervention and automating parts of pathologists' workflow. Specifically, applications of deep learning to…
Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；Broad Institute of Harvard and MIT（哈佛-麻省理工学院博德研究所）；Dana-Farber Cancer Institute（达纳-法伯癌症研究所）
The rapidly emerging field of computational pathology has the potential to enable objective diagnosis, therapeutic response prediction and identification of new morphological features of clinical relevance. However, deep learning-based computational pathology approaches either require manual annotation of gigapixel whole slide images (WSIs) in fully-supervised settings or thousands of WSIs with slide-level labels in a weakly-supervised setting. Moreover, whole slide level computational pathology methods also suffer from domain adaptation and interpretability issues. These challenges have preve…
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；California NanoSystems Institute (CNSI)（加州纳米系统研究所）；David Geffen School of Medicine（大卫·格芬医学院）
Using a deep neural network, we demonstrate a digital staining technique, which we term PhaseStain, to transform quantitative phase images (QPI) of labelfree tissue sections into images that are equivalent to brightfield microscopy images of the same samples that are histochemically stained. Through pairs of image data (QPI and the corresponding brightfield images, acquired after staining) we train a generative adversarial network (GAN) and demonstrate the effectiveness of this virtual staining approach using sections of human skin, kidney and liver tissue, matching the brightfield microscopy …
University of Waterloo（滑铁卢大学）；University of Toronto（多伦多大学）；Huron Digital Pathology（休伦数字病理公司）
Out-of-focus microscopy lens in digital pathology is a critical bottleneck in high-throughput Whole Slide Image (WSI) scanning platforms, for which pixel-level automated Focus Quality Assessment (FQA) methods are highly desirable to help significantly accelerate the clinical workflows. Existing FQA methods include both knowledge-driven and data-driven approaches. While data-driven approaches such as Convolutional Neural Network (CNN) based methods have shown great promises, they are difficult to use in practice due to their high computational complexity and lack of transferability. Here, we pr…
New York University（纽约大学）；Ludwigshafen Hospital（路德维希港医院）；University Hospital Bonn（波恩大学医院）；University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）
Annotation is a major hurdle in the semantic segmentation of microscopy images and volumes due to its prerequisite expertise and effort. This work enables the training of semantic segmentation networks on images with only a single point for training per instance, an extreme case of weak supervision which drastically reduces the burden of annotation. Our approach has two key aspects: (1) we construct a graph-theoretic soft-segmentation using individual seeds to be used within a regularizer during training and (2) we use an objective function that enables learning from the constructed soft-label…
Decision support tools that rely on supervised learning require large amounts of expert annotations. Using past radiological reports obtained from hospital archiving systems has many advantages as training data above manual single-class labels: they are expert annotations available in large quantities, covering a population-representative variety of pathologies, and they provide additional context to pathology diagnoses, such as anatomical location and severity. Learning to auto-generate such reports from images present many challenges such as the difficulty in representing and generating long…
The Norwegian University of Science and Technology（挪威科技大学）；St. Olavs Hospital, Trondheim University Hospital（圣奥拉夫斯医院，特隆赫姆大学医院）；SINTEF
Deep convolutional neural networks (CNNs) are the current state-of-the-art for digital analysis of histopathological images. The large size of whole-slide microscopy images (WSIs) requires advanced memory handling to read, display and process these images. There are several open-source platforms for working with WSIs, but few support deployment of CNN models. These applications use third-party solutions for inference, making them less user-friendly and unsuitable for high-performance image analysis. To make deployment of CNNs user-friendly and feasible on low-end machines, we have developed a …
University of Warwick（华威大学）；Alan Turing Institute（艾伦图灵研究所）；NRP co.（NRP公司）；Tarbiat Modares University（塔比亚特莫达雷斯大学）
Object segmentation is an important step in the workflow of computational pathology. Deep learning based models generally require large amount of labeled data for precise and reliable prediction. However, collecting labeled data is expensive because it often requires expert knowledge, particularly in medical imaging domain where labels are the result of a time-consuming analysis made by one or more human experts. As nuclei, cells and glands are fundamental objects for downstream analysis in computational pathology/cytology, in this paper we propose a simple CNN-based approach to speed up colle…
Unsupervised segmentation of large images using a Potts model Hamiltonian is unique in that segmentation is governed by a resolution parameter which scales the sensitivity to small clusters. Here, the input image is first modeled as a graph, which is then segmented by minimizing a Hamiltonian cost function defined on the graph and the respective segments. However, there exists no closed form solution of this optimization, and using previous iterative algorithmic solution techniques, the problem scales quadratically in the Input Length. Therefore, while Potts model segmentation gives accurate s…
Martinos Center for Biomedical Imaging（马蒂诺斯生物医学成像中心）；Massachusetts General Hospital（麻省总医院）；MGH & BWH Center for Clinical Data Science（麻省总医院与布莱根妇女医院临床数据科学中心）；Mass General Brigham（麻省医疗布列根）；Isomics, Inc.（Isomics公司）；Center for Integrated Diagnostics（综合诊断中心）；Harvard Medical School（哈佛医学院）；PixelMed Publishing, LLC.（PixelMed出版有限责任公司）；Surgical Planning Laboratory（外科规划实验室）；Brigham and Women’s Hospital（布莱根妇女医院）
Machine learning is revolutionizing image-based diagnostics in pathology and radiology. ML models have shown promising results in research settings, but their lack of interoperability has been a major barrier for clinical integration and evaluation. The DICOM a standard specifies Information Object Definitions and Services for the representation and communication of digital images and related information, including image-derived annotations and analysis results. However, the complexity of the standard represents an obstacle for its adoption in the ML community and creates a need for software l…
University of Sydney（悉尼大学）；University of New South Wales（新南威尔士大学）；Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；University of Pittsburgh（匹兹堡大学）；Microsoft Corporation（微软公司）
Unsupervised domain adaptation (UDA) for nuclei instance segmentation is important for digital pathology, as it alleviates the burden of labor-intensive annotation and domain shift across datasets. In this work, we propose a Cycle Consistency Panoptic Domain Adaptive Mask R-CNN (CyC-PDAM) architecture for unsupervised nuclei segmentation in histopathology images, by learning from fluorescence microscopy images. More specifically, we first propose a nuclei inpainting mechanism to remove the auxiliary generated objects in the synthesized images. Secondly, a semantic branch with a domain discrimi…
Google Brain（谷歌大脑）；Google Inc（谷歌公司）；Verily Life Sciences（Verily生命科学）
Each year, the treatment decisions for more than 230,000 breast cancer patients in the U.S. hinge on whether the cancer has metastasized away from the breast. Metastasis detection is currently performed by pathologists reviewing large expanses of biological tissues. This process is labor intensive and error-prone. We present a framework to automatically detect and localize tumors as small as 100 x 100 pixels in gigapixel microscopy images sized 100,000 x 100,000 pixels. Our method leverages a convolutional neural network (CNN) architecture and obtains state-of-the-art results on the Camelyon16…
Sejong University（世宗大学）；University of Warwick（华威大学）；The University of Sheffield（谢菲尔德大学）；Stony Brook University（石溪大学）；National Cancer Institute, National Institutes of Health（美国国立卫生研究院国家癌症研究所）
High-resolution microscopy images of tissue specimens provide detailed information about the morphology of normal and diseased tissue. Image analysis of tissue morphology can help cancer researchers develop a better understanding of cancer biology. Segmentation of nuclei and classification of tissue images are two common tasks in tissue image analysis. Development of accurate and efficient algorithms for these tasks is a challenging problem because of the complexity of tissue morphology and tumor heterogeneity. In this paper we present two computer algorithms; one designed for segmentation of …
Full-color imaging is significant in digital pathology. Compared with a grayscale image or a pseudo-color image that only contains the contrast information, it can identify and detect the target object better with color texture information. Fourier ptychographic microscopy (FPM) is a high-throughput computational imaging technique that breaks the tradeoff between high resolution (HR) and large field-of-view (FOV), which eliminates the artifacts of scanning and stitching in digital pathology and improves its imaging efficiency. However, the conventional full-color digital pathology based on FPM…
Indian Institute of Technology Kharagpur（印度理工学院卡拉格布尔分校）；SigTuple Technologies Private Limited（SigTuple技术私人有限公司）
Adversarially trained deep neural networks have significantly improved performance of single image super resolution, by hallucinating photorealistic local textures, thereby greatly reducing the perception difference between a real high resolution image and its super resolved (SR) counterpart. However, application to medical imaging requires preservation of diagnostically relevant features while refraining from introducing any diagnostically confusing artifacts. We propose using a deep convolutional super resolution network (SRNet) trained for (i) minimising reconstruction loss between the real…
University of Surrey（萨里大学）；National Physical Laboratory（英国国家物理实验室）
The usage of chemical imaging technologies is becoming a routine accompaniment to traditional methods in pathology. Significant technological advances have developed these next generation techniques to provide rich, spatially resolved, multidimensional chemical images. The rise of digital pathology has significantly enhanced the synergy of these imaging modalities with optical microscopy and immunohistochemistry, enhancing our understanding of the biological mechanisms and progression of diseases. Techniques such as imaging mass cytometry provide labelled multidimensional (multiplex) images of…
Computational anatomy allows the quantitative analysis of organs in medical images. However, most analysis is constrained to the millimeter scale because of the limited resolution of clinical computed tomography (CT). X-ray microtomography ($μ$CT) on the other hand allows imaging of ex-vivo tissues at a resolution of tens of microns. In this work, we use clinical CT to image lung cancer patients before partial pneumonectomy (resection of pathological lung tissue). The resected specimen is prepared for $μ$CT imaging at a voxel resolution of 50 $μ$m (0.05 mm). This high-resolution image of the l…
Mapping the landscape of histomorphological cancer phenotypes using self-supervised learning on unlabeled, unannotated pathology slides
University of Glasgow（格拉斯哥大学）；New York University School of Medicine（纽约大学医学院）；New York University Langone Health（纽约大学朗格尼医学中心）；Karolinska Institutet（卡罗林斯卡学院）；University College London Hospital（伦敦大学学院医院）
University of Strasbourg（斯特拉斯堡大学）；CNRS (UMR 7357)（法国国家科学研究中心（UMR 7357））；University of Kragujevac（克拉古耶瓦茨大学）；Hannover Medical School（汉诺威医学院）
Virtual stain transfer is a promising area of research in Computational Pathology, which has a great potential to alleviate important limitations when applying deeplearningbased solutions such as lack of annotations and sensitivity to a domain shift. However, in the literature, the majority of virtual staining approaches are trained for a specific staining or stain combination, and their extension to unseen stainings requires the acquisition of additional data and training. In this paper, we propose HistoStarGAN, a unified framework that performs stain transfer between multiple stainings, stai…
Recent breakthroughs in object detection and image classification using Convolutional Neural Networks (CNNs) are revolutionizing the state of the art in medical imaging, and microscopy in particular presents abundant opportunities for computer vision algorithms to assist medical professionals in diagnosis of diseases ranging from malaria to cancer. High resolution scans of microscopy slides called Whole Slide Images (WSIs) offer enough information for a cancer pathologist to come to a conclusion regarding cancer presence, subtype, and severity based on measurements of features within the slide…
New York University（纽约大学）；IIIT Hyderabad（印度海得拉巴国际信息技术研究所）
Systemic lupus erythematosus (SLE) is an autoimmune disease in which the immune system of the patient starts attacking healthy tissues of the body. Lupus Nephritis (LN) refers to the inflammation of kidney tissues resulting in renal failure due to these attacks. The International Society of Nephrology/Renal Pathology Society (ISN/RPS) has released a classification system based on various patterns observed during renal injury in SLE. Traditional methods require meticulous pathological assessment of the renal biopsy and are time-consuming. Recently, computational techniques have helped to allevi…
Improving Colorectal Cancer Screening and Risk Assessment through Predictive Modeling on Medical Images and Records
Geisel School of Medicine at Dartmouth（达特茅斯盖泽尔医学院）；Dartmouth-Hitchcock Medical Center（达特茅斯-希区柯克医学中心）；New Hampshire Colonoscopy Registry（新罕布什尔结肠镜检查登记处）；White River Junction VA Medical Center（怀特河交界处退伍军人事务医疗中心）；Dartmouth College（达特茅斯学院）
Geisel School of Medicine at Dartmouth（达特茅斯盖泽尔医学院）；Dartmouth College（达特茅斯学院）；Dartmouth-Hitchcock Medical Center（达特茅斯-希契科克医疗中心）
Deep learning-based methods, such as the sliding window approach for cropped-image classification and heuristic aggregation for whole-slide inference, for analyzing histological patterns in high-resolution microscopy images have shown promising results. These approaches, however, require a laborious annotation process and are fragmented. This diagnostic study collected deidentified high-resolution histological images (N = 379) for training a new model composed of a convolutional neural network and a grid-based attention network, trainable without region-of-interest annotations. Histological im…
Stanford University School of Medicine（斯坦福大学医学院）；Stanford University（斯坦福大学）；Bonn University Hospital（波恩大学医院）；Institute for Pathology and Cytology（病理与细胞学研究所）；University Hospital Muenster（明斯特大学医院）
The current study detects different morphologies related to prostate pathology using deep learning models; these models were evaluated on 2,121 hematoxylin and eosin (H&E) stain histology images captured using bright field microscopy, which spanned a variety of image qualities, origins (whole slide, tissue micro array, whole mount, Internet), scanning machines, timestamps, H&E staining protocols, and institutions. For case usage, these models were applied for the annotation tasks in clinician-oriented pathology reports for prostatectomy specimens. The true positive rate (TPR) for slides with p…
Deep-learning-based pipelines have shown the potential to revolutionalize microscopy image diagnostics by providing visual augmentations to a trained pathology expert. However, to match human performance, the methods rely on the availability of vast amounts of high-quality labeled data, which poses a significant challenge. To circumvent this, augmented labeling methods, also known as expert-algorithm-collaboration, have recently become popular. However, potential biases introduced by this operation mode and their effects for training neuronal networks are not entirely understood. This work aim…
Towards a Comprehensive Benchmark for Pathological Lymph Node Metastasis in Breast Cancer Sections
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital & Shenzhen Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College（中国医学科学院北京协和医学院肿瘤医院（国家癌症中心/国家临床肿瘤医学中心）深圳医院）；Research Institute of Tsinghua, Guangzhou（清华大学广州研究院）
Model-agnostic Knowledge Augmented Machine Learning for Weakly Annotated Bioimaging Data
The University of Melbourne（墨尔本大学）；University of Wollongong（伍伦贡大学）；The University of Sydney（悉尼大学）；The Florey Institute of Neuroscience and Mental Health, The University of Melbourne（墨尔本大学弗洛里神经科学与心理健康研究所）；GenieUs Genomics Pty Ltd（GenieUs Genomics有限公司）；Xstendyl
University of California, Los Angeles（加州大学洛杉矶分校）；California NanoSystems Institute (CNSI)（加州纳米系统研究所）；David Geffen School of Medicine（戴维·格芬医学院）
Pathological diagnosis relies on the visual inspection of histologically stained thin tissue specimens, where different types of stains are applied to bring contrast to and highlight various desired histological features. However, the destructive histochemical staining procedures are usually irreversible, making it very difficult to obtain multiple stains on the same tissue section. Here, we demonstrate a virtual stain transfer framework via a cascaded deep neural network (C-DNN) to digitally transform hematoxylin and eosin (H&E) stained tissue images into other types of histological stains. U…
Institute of Biophysics, National Research Council of Italy（意大利国家研究委员会生物物理研究所）；Institute of Information Science and Technologies, National Research Council of Italy（意大利国家研究委员会信息科学与技术研究所）
Early diagnosis of cancer often allows for a more vast choice of therapy opportunities. After a cancer diagnosis, staging provides essential information about the extent of disease in the body and the expected response to a particular treatment. The leading importance of classifying cancer patients at the early stage into high or low-risk groups has led many research teams, both from the biomedical and bioinformatics field, to study the application of Deep Learning (DL) methods. The ability of DL to detect critical features from complex datasets is a significant achievement in early diagnosis …
Taiyuan University of Technology（太原理工大学）；University of Science and Technology of China（中国科学技术大学）；Second Hospital of Shanxi Medical University（山西医科大学第二医院）
We report an angle-tilted, wavelength-multiplexed ptychographic modulation approach for multispectral lensless on-chip microscopy. In this approach, we illuminate the specimen with lights at 5 wavelengths simultaneously. A prism is added at the illumination path for spectral dispersion. Lightwaves at different wavelengths, thus, hit the specimen at slightly different incident angles, breaking the ambiguities in mixed state ptychographic reconstruction. At the detection path, we place a thin diffuser in-between the specimen and the monochromatic image sensor for encoding the spectral informatio…
University of California, Los Angeles（加州大学洛杉矶分校）；California NanoSystems Institute（加州纳米系统研究所）；David Geffen School of Medicine（大卫·格芬医学院）
Polarized light microscopy provides high contrast to birefringent specimen and is widely used as a diagnostic tool in pathology. However, polarization microscopy systems typically operate by analyzing images collected from two or more light paths in different states of polarization, which lead to relatively complex optical designs, high system costs or experienced technicians being required. Here, we present a deep learning-based holographic polarization microscope that is capable of obtaining quantitative birefringence retardance and orientation information of specimen from a phase recovered …
Current pathology workflow involves staining of thin tissue slices, which otherwise would be transparent, followed by manual investigation under the microscope by a trained pathologist. While the hematoxylin and eosin (H&E) stain is well-established and a cost-effective method for visualizing histology slides, its color variability across preparations and subjectivity across clinicians remain unaddressed challenges. To mitigate these challenges, recently we have demonstrated that spatial light interference microscopy (SLIM) can provide a path to intrinsic, objective markers, that are independe…
INESC TEC - Institute for Systems and Computer Engineering, Technology and Science（系统与计算机工程、技术与科学研究所）；Faculty of Engineering of University of Porto（波尔图大学工程学院）；Seek AI Limited（Seek AI有限公司）；Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院）；The Mount Sinai Hospital（西奈山医院）
Breast cancer is the most common invasive cancer in women, affecting more than 10% of women worldwide. Microscopic analysis of a biopsy remains one of the most important methods to diagnose the type of breast cancer. This requires specialized analysis by pathologists, in a task that i) is highly time- and cost-consuming and ii) often leads to nonconsensual results. The relevance and potential of automatic classification algorithms using hematoxylin-eosin stained histopathological images has already been demonstrated, but the reported results are still sub-optimal for clinical use. With the goa…
Introducing NCL-SM: A Fully Annotated Dataset of Images from Human Skeletal Muscle Biopsies
Wellcome Centre for Mitochondrial Research（惠康线粒体研究中心）；Translational and Clinical Research Institute（转化与临床研究所）；Faculty of Medical Sciences（医学院）；Newcastle University（纽卡斯尔大学）；School of Computing（计算机学院）
Fourier ptychographic microscopy (FPM) is a computational approach geared towards creating high-resolution and large field-of-view images without mechanical scanning. To acquire color images of histology slides, it often requires sequential acquisitions with red, green, and blue illuminations. The color reconstructions often suffer from coherent artifacts that are not presented in regular incoherent microscopy images. As a result, it remains a challenge to employ FPM for digital pathology applications, where resolution and color accuracy are of critical importance. Here we report a deep learni…
Histo-Miner: Deep learning based tissue features extraction pipeline from H&E whole slide images of cutaneous squamous cell carcinoma
Faculty of Mathematics and Natural Sciences, University of Cologne（数学与自然科学学院，科隆大学）；Excellence Cluster on Cellular Stress Responses in Aging-Associated Diseases (CECAD), University of Cologne（细胞应激反应与年龄相关疾病卓越集群（CECAD），科隆大学）；Department for Dermatology, University Hospital Cologne（皮肤科部门，科隆大学医院）；Department of Dermatology and Allergy, School of Medicine, Technical University of Munich（皮肤科与过敏科部门，医学院，慕尼黑技术大学）；Department of Dermatology, Dortmund Hospital gGmbH, University Witten/Herdecke（皮肤科部门，多特蒙德医院gGmbH，乌尔姆/赫尔德克大学）；Department of Dermatology, Venereology and Allergology, Helios St. Elisabeth Hospital Oberhausen, University Witten/Herdecke（皮肤科、性病科与过敏科部门，Helios圣埃利莎医院奥伯豪森，乌尔姆/赫尔德克大学）；Department of Dermatology and Allergology, University Hospital of the Paracelsus Medical University Salzburg（皮肤科与过敏科部门，帕拉塞尔医学大学萨尔茨堡大学医院）；Department of Dermatology and Allergology, University Hospital Bonn（皮肤科与过敏科部门，波恩大学医院）；Medical Clinic III for Oncology, Hematology, Immune-Oncology and Rheumatology, University Hospital Bonn (UKB)（肿瘤科、血液科、免疫肿瘤科和风湿科第三医疗部门，波恩大学医院（UKB））
