Counting Cells in Time-Lapse Microscopy using Deep Neural Networks
Istituto Sistemi e Electronica Applicata (ISEA)（应用系统与电子研究所）；Scuola universitaria professionale della Svizzera italiana (SUPSI)（瑞士意大利语区应用科技大学）；Grupo de investigacion SISTEMIC（SISTEMIC研究组）；Facultad de Ingenieria, Universidad de Antioquia UdeA（安蒂奥基亚大学工程学院）
Alexander Gomez Villa, Augusto Salazar, Igor Stefanini
An automatic approach to counting any kind of cells could alleviate work of the experts and boost the research in fields such as regenerative medicine. In this paper, a method for microscopy cell counting using multiple frames (hence temporal information) is proposed. Unlike previous approaches where the cell counting is done independently in each frame (static cell counting), in this work the cell counting prediction is done using multiple frames (dynamic cell counting). A spatiotemporal model using ConvNets and long short term memory (LSTM) recurrent neural networks is proposed to overcome temporal variations. The model outperforms static cell counting in a publicly available dataset of stem cells. The advantages, working conditions and limitations of the ConvNet-LSTM method are discussed. Although our method is tested in cell counting, it can be extrapolated to quantify in video (or correlated image series) any kind of objects or volumes.
The University of Adelaide（阿德莱德大学）；The Australian National University（澳大利亚国立大学）；Curtin University of Technology（科廷科技大学）；CSIRO（澳大利亚联邦科学与工业研究组织）
Quantitative analysis of the dynamics of tiny cellular and sub-cellular structures, known as particles, in time-lapse cell microscopy sequences requires the development of a reliable multi-target tracking method capable of tracking numerous similar targets in the presence of high levels of noise, high target density, complex motion patterns and intricate interactions. In this paper, we propose a framework for tracking these structures based on the random finite set Bayesian filtering framework. We focus on challenging biological applications where image characteristics such as noise and backgr…
Time-lapse fluorescent microscopy (TLFM) combined with predictive mathematical modelling is a powerful tool to study the inherently dynamic processes of life on the single-cell level. Such experiments are costly, complex and labour intensive. A complimentary approach and a step towards in silico experimentation, is to synthesise the imagery itself. Here, we propose Multi-StyleGAN as a descriptive approach to simulate time-lapse fluorescence microscopy imagery of living cells, based on a past experiment. This novel generative adversarial network synthesises a multi-domain sequence of consecutiv…
Accurate Tracking of Arabidopsis Root Cortex Cell Nuclei in 3D Time-Lapse Microscopy Images Based on Genetic Algorithm
Graduate School of Science and Technology, Nara Institute of Science and Technology（奈良科学技术大学研究生院）；Bioresource Engineering Laboratory, Ishikawa Prefectural University（石川县立大学生物资源工程实验室）；Department of Engineering Informatics, Osaka Electro-Communication University（大阪电讯大学工程信息学系）；Exploratory Research Center on Life and Living Systems, National Institutes of Natural Sciences（国家自然科学研究院生命与生活系统探索研究中心）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）
University of Bologna（博洛尼亚大学）；University of Tor Vergata（罗马第二大学（托尔·韦尔加塔大学））；Institute Curie Centre de Recherche（居里研究所研究中心）；Paris Sciences et Lettres Research University（巴黎文理研究大学）
Biological experiments based on organ-on-chips (OOCs) exploit light Time-Lapse Microscopy (TLM) for a direct observation of cell movement that is an observable signature of underlying biological processes. A high spatial resolution is essential to capture cell dynamics and interactions from recorded experiments by TLM. Unfortunately, due to physical and cost limitations, acquiring high resolution videos is not always possible. To overcome the problem, we present here a new deep learning-based algorithm that extends the well known Deep Image Prior (DIP) to TLM Video Super Resolution (SR) withou…
University of California Santa Barbara（加州大学圣巴巴拉分校）
We propose a novel weakly supervised method to improve the boundary of the 3D segmented nuclei utilizing an over-segmented image. This is motivated by the observation that current state-of-the-art deep learning methods do not result in accurate boundaries when the training data is weakly annotated. Towards this, a 3D U-Net is trained to get the centroid of the nuclei and integrated with a simple linear iterative clustering (SLIC) supervoxel algorithm that provides better adherence to cluster boundaries. To track these segmented nuclei, our algorithm utilizes the relative nuclei location depict…
Cell detection is the task of detecting the approximate positions of cell centroids from microscopy images. Recently, convolutional neural network-based approaches have achieved promising performance. However, these methods require a certain amount of annotation for each imaging condition. This annotation is a time-consuming and labor-intensive task. To overcome this problem, we propose a semi-supervised cell-detection method that effectively uses a time-lapse sequence with one labeled image and the other images unlabeled. First, we train a cell-detection network with a one-labeled image and e…
State University of New York at Stony Brook（纽约州立大学石溪分校）
Microscopy images from different imaging conditions, organs, and tissues often have numerous cells with various shapes on a range of backgrounds. As a result, designing a deep learning model to count cells in a source domain becomes precarious when transferring them to a new target domain. To address this issue, manual annotation costs are typically the norm when training deep learning-based cell counting models across different domains. In this paper, we propose a cross-domain cell counting approach that requires only weak human annotation efforts. Initially, we implement a cell counting netw…
Cell Behavior Video Classification Challenge, a benchmark for computer vision methods in time-lapse microscopy
Euler Institute, Faculty of Informatics, Università della Svizzera italiana（欧拉研究所，信息学院，瑞士大学）；International Center for Advanced Computing in Medicine (ICAM), University of Pavia（国际医学先进计算中心（ICAM），帕维亚大学）；Imaging Platform, ACRF INCITe Centre, Garvan Institute of Medical Research（成像平台，ACRF INCITe中心，嘉文医学研究所）；Tacoma School of Engineering & Technology, University of Washington（塔科马工程与技术学院，华盛顿大学）；Data Science, Institute for Computing and Information Sciences, Radboud University（数据科学，计算与信息科学研究所，拉德堡德大学）；Imaging Core, Lerner Research Institute, Cleveland Clinic（成像核心，勒纳研究研究所，克利夫兰诊所）；Institute of Biomedical Engineering, University of Toronto（生物医学工程研究所，多伦多大学）；Center for Research in Computer Vision, University of Central Florida（计算机视觉研究中心，佛罗里达大学）；Computational Biology Group, Data Science Platform, Garvan Institute of Medical Research（计算生物学小组，数据科学平台，嘉文医学研究所）
New York University（纽约大学）；University Hospital Bonn（波恩大学医院）
Object counting and localization are key steps for quantitative analysis in large-scale microscopy applications. This procedure becomes challenging when target objects are overlapping, are densely clustered, and/or present fuzzy boundaries. Previous methods producing density maps based on deep learning have reached a high level of accuracy for object counting by assuming that object counting is equivalent to the integration of the density map. However, this model fails when objects show significant overlap regarding accurate localization. We propose an alternative method to count and localize …
CellFMCount: A Fluorescence Microscopy Dataset, Benchmark, and Methods for Cell Counting
Department of Computer Science（计算机科学系）；Department of Genetics, Development, and Cell Biology（遗传学、发育与细胞生物学系）；Molecular, Cellular, and Developmental Biology Program（分子、细胞与发育生物学项目）；Neuroscience Program（神经科学项目）；Nanovaccine Institute（纳米疫苗研究所）；Iowa State University（爱荷华州立大学）
Cell counting is a ubiquitous, yet tedious task that would greatly benefit from automation. From basic biological questions to clinical trials, cell counts provide key quantitative feedback that drive research. Unfortunately, cell counting is most commonly a manual task and can be time-intensive. The task is made even more difficult due to overlapping cells, existence of multiple focal planes, and poor imaging quality, among other factors. Here, we describe a convolutional neural network approach, using a recently described feature pyramid network combined with a VGG-style neural network, for …
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Carle Foundation Hospital（卡尔基金会医院）
Accurately counting the number of cells in microscopy images is required in many medical diagnosis and biological studies. This task is tedious, time-consuming, and prone to subjective errors. However, designing automatic counting methods remains challenging due to low image contrast, complex background, large variance in cell shapes and counts, and significant cell occlusions in two-dimensional microscopy images. In this study, we proposed a new density regression-based method for automatically counting cells in microscopy images. The proposed method processes two innovations compared to othe…
University of South Bohemia in České Budějovice（南波希米亚大学）
This article presents an algorithm for the evaluation of organelles' movements inside of an unmodified live cell. We used a time-lapse image series obtained using wide-field bright-field photon transmission microscopy as an algorithm input. The benefit of the algorithm is the application of the Rényi information entropy, namely a variable called a point information gain, which enables to highlight the borders of the intracellular organelles and to localize the organelles' centers of mass with the precision of one pixel.
Institute of Physics of the Czech Academy of Sciences, v.v.i.（捷克科学院物理研究所）；Regional Centre of Advanced Technologies and Materials, Palacký University（帕拉茨基大学先进技术与材料区域中心）；RWTH Aachen University（亚琛工业大学）
Scanning tunneling and atomic force microscopies (STM/nc-AFM) are rapidly progressing to offer unprecedented spatial resolution of a diverse array of chemical species. In particular, they are employed to characterize on-surface chemical reactions by directly examining precursors and products. Chiral effects and self-assembled structures can also be investigated. This open source, modular, python based scheme automates the categorization of a variety of molecules present in medium sized (10$\times$10 to 100$\times$100 nm) scanned probe images.
Kymographs are widely used to represent and anal- yse spatio-temporal dynamics of fluorescence markers along curvilinear biological compartments. These objects have a sin- gular geometry, thus kymograph reconstruction is inherently an analog image processing task. However, the existing approaches are essentially digital: the kymograph photometry is sampled directly from the time-lapse images. As a result, such kymographs rely on raw image data that suffer from the degradations entailed by the image formation process and the spatio-temporal resolution of the imaging setup. In this work, we addr…
Time-lapse fluorescence microscopy (TLFM) is an important and powerful tool in synthetic biological research. Modeling TLFM experiments based on real data may enable researchers to repeat certain experiments with minor effort. This thesis is a study towards deep learning-based modeling of TLFM experiments on the image level. The modeling of TLFM experiments, by way of the example of trapped yeast cells, is split into two tasks. The first task is to generate synthetic image data based on real image data. To approach this problem, a novel generative adversarial network, for conditionalized and u…
We present a novel graph neural network (GNN) approach for cell tracking in high-throughput microscopy videos. By modeling the entire time-lapse sequence as a direct graph where cell instances are represented by its nodes and their associations by its edges, we extract the entire set of cell trajectories by looking for the maximal paths in the graph. This is accomplished by several key contributions incorporated into an end-to-end deep learning framework. We exploit a deep metric learning algorithm to extract cell feature vectors that distinguish between instances of different biological cells…
Istituto Nazionale di Fisica Nucleare（国家核物理研究所）；University of Bologna（博洛尼亚大学）
Counting cells in fluorescent microscopy is a tedious, time-consuming task that researchers have to accomplish to assess the effects of different experimental conditions on biological structures of interest. Although such objects are generally easy to identify, the process of manually annotating cells is sometimes subject to arbitrariness due to the operator's interpretation of the borderline cases. We propose a Machine Learning approach that exploits a fully-convolutional network in a binary segmentation fashion to localize the objects of interest. Counts are then retrieved as the number of d…
Deep Neural Networks are increasingly used in video frame interpolation tasks such as frame rate changes as well as generating fake face videos. Our project aims to apply recent advances in Deep video interpolation to increase the temporal resolution of fluorescent microscopy time-lapse movies. To our knowledge, there is no previous work that uses Convolutional Neural Networks (CNN) to generate frames between two consecutive microscopy images. We propose a fully convolutional autoencoder network that takes as input two images and generates upto seven intermediate images. Our architecture has t…
University of Vienna（维也纳大学）；Radon Institute of Computational and Applied Mathematics, Austrian Academy of Sciences（奥地利科学院拉东计算与应用数学研究所）
In this work we consider optical flow on evolving Riemannian 2-manifolds which can be parametrised from the 2-sphere. Our main motivation is to estimate cell motion in time-lapse volumetric microscopy images depicting fluorescently labelled cells of a live zebrafish embryo. We exploit the fact that the recorded cells float on the surface of the embryo and allow for the extraction of an image sequence together with a sphere-like surface. We solve the resulting variational problem by means of a Galerkin method based on vector spherical harmonics and present numerical results computed from the af…
Department of Molecular and Cellular Biology（分子与细胞生物学系）；School of Engineering and Applied Sciences（工程与应用科学学院）；Tel Aviv Sourasky Medical Center（特拉维夫苏拉斯基医疗中心）；Harvard Graduate Program in Biophysics（哈佛大学生物物理学研究生项目）；Harvard University（哈佛大学）
A major challenge in clinical In-Vitro Fertilization (IVF) is selecting the highest quality embryo to transfer to the patient in the hopes of achieving a pregnancy. Time-lapse microscopy provides clinicians with a wealth of information for selecting embryos. However, the resulting movies of embryos are currently analyzed manually, which is time consuming and subjective. Here, we automate feature extraction of time-lapse microscopy of human embryos with a machine-learning pipeline of five convolutional neural networks (CNNs). Our pipeline consists of (1) semantic segmentation of the regions of …
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；California NanoSystems Institute（加州纳米系统研究所）；Istanbul University（伊斯坦布尔大学）；David Geffen School of Medicine（大卫·格芬医学院）
We present a computational live bacteria detection system that periodically captures coherent microscopy images of bacterial growth inside a 60 mm diameter agar-plate and analyzes these time-lapsed holograms using deep neural networks for rapid detection of bacterial growth and classification of the corresponding species. The performance of our system was demonstrated by rapid detection of Escherichia coli and total coliform bacteria (i.e., Klebsiella aerogenes and Klebsiella pneumoniae subsp. pneumoniae) in water samples. These results were confirmed against gold-standard culture-based result…
Slovak University of Technology in Bratislava（布拉迪斯拉发斯洛伐克理工大学）；CNRS（法国国家科学研究中心）；Univ. Montpellier（蒙彼利埃大学）
The automated segmentation and tracking of macrophages during their migration are challenging tasks due to their dynamically changing shapes and motions. This paper proposes a new algorithm to achieve automatic cell tracking in time-lapse microscopy macrophage data. First, we design a segmentation method employing space-time filtering, local Otsu's thresholding, and the SUBSURF (subjective surface segmentation) method. Next, the partial trajectories for cells overlapping in the temporal direction are extracted in the segmented images. Finally, the extracted trajectories are linked by consideri…
University of Tennessee Health Science Center（田纳西大学健康科学中心）
The ability to automatically detect, classify, calculate the size, number, and grade of retinal cells and other biological objects is critically important in eye disease like age-related macular degeneration (AMD). In this paper, we developed an automated tool based on deep learning technique and Mask R-CNN model to analyze large datasets of transmission electron microscopy (TEM) images and quantify retinal cells with high speed and precision. We considered three categories for outer nuclear layer (ONL) cells: live, intermediate, and pyknotic. We trained the model using a dataset of 24 samples…
Unsupervised cell segmentation by fast Gaussian Processes
Department of Statistics and Applied Probability, University of California, Santa Barbara, CA（统计与应用概率系，加州大学圣芭芭拉分校）；Department of Electrical and Computer Engineering, University of California, Santa Barbara, CA（电气与计算机工程系，加州大学圣芭芭拉分校）
Oxford Brookes University（牛津布鲁克斯大学）；Vellore Institute of Technology（韦洛尔理工学院）
Tuberculosis is a contagious disease which is one of the leading causes of death, globally. The general diagnosis methods for tuberculosis include microscopic examination, tuberculin skin test, culture method, enzyme linked immunosorbent assay (ELISA) and electronic nose system. World Health Organization (WHO) recommends standard microscopic examination for early diagnosis of tuberculosis. In microscopy, the technician examines field of views (FOVs) in sputum smear for presence of any TB bacilli and counts the number of TB bacilli per FOV to report the level of severity. This process is time c…
Topological texture analysis of microscopy images of dynamic casein gelation and its relation to rheological properties
Department of Computer Science, University of Copenhagen, Denmark（哥本哈根大学计算机科学系）；Department of Green Technology, University of Southern Denmark, Denmark（南丹麦大学绿色技术系）；Department of Food Science, University of Copenhagen, Denmark（哥本哈根大学食品科学系）
In this work we consider brightness and mass conservation laws for motion estimation on evolving Riemannian 2-manifolds that allow for a radial parametrisation from the 2-sphere. While conservation of brightness constitutes the foundation for optical flow methods and has been generalised to said scenario, we formulate in this article the principle of mass conservation for time-varying surfaces which are embedded in Euclidean 3-space and derive a generalised continuity equation. The main motivation for this work is efficient cell motion estimation in time-lapse (4D) volumetric fluorescence micr…
Deep Joint Denoising and Detection for Enhanced Intracellular Particle Analysis
Erasmus University Medical Center（伊拉斯姆斯大学医学中心）；Utrecht University（乌得勒支大学）；University of New South Wales (UNSW)（新南威尔士大学）；NVIDIA Corporation（英伟达公司）；SURFsara
Clear cell renal cell carcinoma (ccRCC) is one of the most common forms of intratumoral heterogeneity in the study of renal cancer. ccRCC originates from the epithelial lining of proximal convoluted renal tubules. These cells undergo abnormal mutations in the presence of Ki67 protein and create a lump-like structure through cell proliferation. Manual counting of tumor cells in the tissue-affected sections is one of the strongest prognostic markers for renal cancer. However, this procedure is time-consuming and also prone to subjectivity. These assessments are based on the physical cell appeara…
Laboratoire Jean-Perrin UMR 8237, Sorbonne Université（索邦大学让-佩兰实验室）；SABILAb；CMAP, Ecole Polytechnique, Université Paris-Saclay（巴黎萨克雷大学综合理工学院应用数学中心）；Heuritech
The mother machine is a popular microfluidic device that allows long-term time-lapse imaging of thousands of cells in parallel by microscopy. It has become a valuable tool for single-cell level quantitative analysis and characterization of many cellular processes such as gene expression and regulation, mutagenesis or response to antibiotics. The automated and quantitative analysis of the massive amount of data generated by such experiments is now the limiting step. In particular the segmentation and tracking of bacteria cells imaged in phase-contrast microscopy---with error rates compatible wi…
Dr. A.P.J. Abdul Kalam Technical University（阿卜杜勒·卡拉姆技术大学）
Every year millions of people die due to disease of Cancer. Due to its invasive nature it is very complex to cure even in primary stages. Hence, only method to survive this disease completely is via forecasting by analyzing the early mutation in cells of the patient biopsy. Cell Segmentation can be used to find cell which have left their nuclei. This enables faster cure and high rate of survival. Cell counting is a hard, yet tedious task that would greatly benefit from automation. To accomplish this task, segmentation of cells need to be accurate. In this paper, we have improved the learning o…
Automated Palynological Analysis System: Integrating Deep Metric Learning, Detection and Classification in Bright Field Microscopy
Departamento de Física, Facultad de Ciencias Físicas y Matemáticas, Universidad de Concepción（物理系，物理与数学科学学院，康塞普西翁大学）；Facultad de Ingeniería, Universidad San Sebastián（工程学院，圣塞巴斯蒂安大学）；Facultad de Ingeniería, Universidad Andres Bello（工程学院，安德烈斯·贝洛大学）；Universidad Arturo Prat, Sede Victoria（阿尔托普大学，维多利亚校区）；Laboratory of Palynology and Plant Ecology, School of Sciences and Technologies, University of Concepción（古生物与植物生态实验室，科学与技术学院，康塞普西翁大学）
Fluorescent Neuronal Cells v2: Multi-Task, Multi-Format Annotations for Deep Learning in Microscopy
National Institute of Nuclear Physics（国家核物理研究所）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））；University of Bologna（博洛尼亚大学）
