Imaging assays of cellular function, especially those using fluorescent stains, are ubiquitous in the biological and medical sciences. Despite advances in computer vision, such images are often analyzed using only manual or rudimentary automated processes. Watershed-based segmentation is an effective technique for identifying objects in images; it outperforms commonly used image analysis methods, but requires familiarity with computer-vision techniques to be applied successfully. In this report, we present and implement a watershed-based image analysis and classification algorithm in a GUI, enabling a broad set of users to easily understand the algorithm and adjust the parameters to their specific needs. As an example, we implement this algorithm to find and classify cells in a complex imaging assay for mitochondrial function. In a second example, we demonstrate a workflow using manual comparisons and receiver operator characteristics to optimize the algorithm parameters for finding live and dead cells in a standard viability assay. Overall, this watershed-based algorithm is more advanced than traditional thresholding and can produce optimized, automated results. By incorporating associated pre-processing steps in the GUI, the algorithm is also easily adjusted, rendering it user-friendly.
RWTH Aachen University（亚琛工业大学）；Brazilian Center for Research in Energy and Materials（巴西能源与材料研究中心）；Brazilian Synchrotron Light Laboratory（巴西同步加速器光实验室）；Howard Hughes Medical Institute（霍华德·休斯医学研究所）
The quantitative analysis of cellular membranes helps understanding developmental processes at the cellular level. Particularly 3D microscopic image data offers valuable insights into cell dynamics, but error-free automatic segmentation remains challenging due to the huge amount of data generated and strong variations in image intensities. In this paper, we propose a new 3D segmentation approach which combines the discriminative power of convolutional neural networks (CNNs) for preprocessing and investigates the performance of three watershed-based postprocessing strategies (WS), which are wel…
In recent years, an enormous amount of fluorescence microscopy images were collected in high-throughput lab settings. Analyzing and extracting relevant information from all images in a short time is almost impossible. Detecting tiny individual cell compartments is one of many challenges faced by biologists. This paper aims at solving this problem by building an end-to-end process that employs methods from the deep learning field to automatically segment, detect and classify cell compartments of fluorescence microscopy images of yeast cells. With this intention we used Mask R-CNN to automatical…
Max Planck Institute of Molecular Cell Biology and Genetics（马克斯·普朗克分子细胞生物学与遗传学研究所）；Center for Systems Biology Dresden（德累斯顿系统生物学中心）；Faculty of Computer Science, Technical University Dresden（德累斯顿工业大学计算机科学系）
Fluorescence microscopy images usually show severe anisotropy in axial versus lateral resolution. This hampers downstream processing, i.e. the automatic extraction of quantitative biological data. While deconvolution methods and other techniques to address this problem exist, they are either time consuming to apply or limited in their ability to remove anisotropy. We propose a method to recover isotropic resolution from readily acquired anisotropic data. We achieve this using a convolutional neural network that is trained end-to-end from the same anisotropic body of data we later apply the net…
Compressed sensing fluorescence microscopy (CS-FM) proposes a scheme whereby less measurements are collected during sensing and reconstruction is performed to recover the image. Much work has gone into optimizing the sensing and reconstruction portions separately. We propose a method of jointly optimizing both sensing and reconstruction end-to-end under a total measurement constraint, enabling learning of the optimal sensing scheme concurrently with the parameters of a neural network-based reconstruction network. We train our model on a rich dataset of confocal, two-photon, and wide-field micr…
Czech Academy of Sciences（捷克科学院）；Institute of Information Theory and Automation（信息理论与自动化研究所）；Washington University in St. Louis（圣路易斯华盛顿大学）；Washington University School of Medicine（华盛顿大学医学院）
Three-dimensional fluorescence microscopy often suffers from anisotropy, where the resolution along the axial direction is lower than that within the lateral imaging plane. We address this issue by presenting Dual-Cycle, a new framework for joint deconvolution and fusion of dual-view fluorescence images. Inspired by the recent Neuroclear method, Dual-Cycle is designed as a cycle-consistent generative network trained in a self-supervised fashion by combining a dual-view generator and prior-guided degradation model. We validate Dual-Cycle on both synthetic and real data showing its state-of-the-…
University of Michigan-Shanghai Jiao Tong University Joint Institute（上海交通大学密西根学院）；Shanghai Jiao Tong University（上海交通大学）
Fluorescence microscopy plays an important role in biomedical research. The depth-variant point spread function (PSF) of a fluorescence microscope produces low-quality images especially in the out-of-focus regions of thick specimens. Traditional deconvolution to restore the out-of-focus images is usually insufficient since a depth-invariant PSF is assumed. This article aims at handling fluorescence microscopy images by learning-based depth-variant PSF and reducing artifacts. We propose adaptive weighting depth-variant deconvolution (AWDVD) with defocus level prediction convolutional neural net…
Fluorescence microscopy has enabled a dramatic development in modern biology. Due to its inherently weak signal, fluorescence microscopy is not only much noisier than photography, but also presented with Poisson-Gaussian noise where Poisson noise, or shot noise, is the dominating noise source. To get clean fluorescence microscopy images, it is highly desirable to have effective denoising algorithms and datasets that are specifically designed to denoise fluorescence microscopy images. While such algorithms exist, no such datasets are available. In this paper, we fill this gap by constructing a …
Robust and accurate nuclei centroid detection is important for the understanding of biological structures in fluorescence microscopy images. Existing automated nuclei localization methods face three main challenges: (1) Most of object detection methods work only on 2D images and are difficult to extend to 3D volumes; (2) Segmentation-based models can be used on 3D volumes but it is computational expensive for large microscopy volumes and they have difficulty distinguishing different instances of objects; (3) Hand annotated ground truth is limited for 3D microscopy volumes. To address these iss…
Fluorescence microscopy has become a widely used tool for studying various biological structures of in vivo tissue or cells. However, quantitative analysis of these biological structures remains a challenge due to their complexity which is exacerbated by distortions caused by lens aberrations and light scattering. Moreover, manual quantification of such image volumes is an intractable and error-prone process, making the need for automated image analysis methods crucial. This paper describes a segmentation method for tubular structures in fluorescence microscopy images using convolutional neura…
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；California Nano Systems Institute (CNSI)（加州纳米系统研究所）；David Geffen School of Medicine（大卫·格芬医学院）；Hebrew University of Jerusalem（耶路撒冷希伯来大学）
Three-dimensional (3D) fluorescence microscopy in general requires axial scanning to capture images of a sample at different planes. Here we demonstrate that a deep convolutional neural network can be trained to virtually refocus a 2D fluorescence image onto user-defined 3D surfaces within the sample volume. With this data-driven computational microscopy framework, we imaged the neuron activity of a Caenorhabditis elegans worm in 3D using a time-sequence of fluorescence images acquired at a single focal plane, digitally increasing the depth-of-field of the microscope by 20-fold without any axi…
Fluorescence microscopy is a quintessential tool for observing cells and understanding the underlying mechanisms of life-sustaining processes of all living organisms. The problem of extracting 3D shape of mitochondria from fluorescence microscopy images remains unsolved due to the complex and varied shapes expressed by mitochondria and the poor resolving capacity of these microscopes. We propose an approach to bridge this gap by learning a shape prior for mitochondria termed as MiShape, by leveraging high-resolution electron microscopy data. MiShape is a generative model learned using implicit…
Multiple stains are usually used to highlight biological substances in biomedical image analysis. To decompose multiple stains for co-localization quantification, blind source separation is usually performed. Prior model-based stain separation methods usually rely on stains' spatial distributions over an image and may fail to solve the co-localization problem. With the advantage of machine learning, deep generative models are used for this purpose. Since prior knowledge of imaging models is ignored in purely data-driven solutions, these methods may be sub-optimal. In this study, a novel learni…
This paper introduces a new method of data-driven microscope design for virtual fluorescence microscopy. Our results show that by including a model of illumination within the first layers of a deep convolutional neural network, it is possible to learn task-specific LED patterns that substantially improve the ability to infer fluorescence image information from unstained transmission microscopy images. We validated our method on two different experimental setups, with different magnifications and different sample types, to show a consistent improvement in performance as compared to conventional…
Saudi Electronic University（沙特电子大学）；The Commonwealth Scientific and Industrial Research Organisation（澳大利亚联邦科学与工业研究组织）；The Australian National University（澳大利亚国立大学）；University of Technology Sydney（悉尼科技大学）；the University of Western Australia（西澳大利亚大学）
Accurate localization of proteins from fluorescence microscopy images is challenging due to the inter-class similarities and intra-class disparities introducing grave concerns in addressing multi-class classification problems. Conventional machine learning-based image prediction pipelines rely heavily on pre-processing such as normalization and segmentation followed by hand-crafted feature extraction to identify useful, informative, and application-specific features. Here, we demonstrate that deep learning-based pipelines can effectively classify protein images from different datasets. We prop…
The trade-off between throughput and image quality is an inherent challenge in microscopy. To improve throughput, compressive imaging under-samples image signals; the images are then computationally reconstructed by solving a regularized inverse problem. Compared to traditional regularizers, Deep Learning based methods have achieved greater success in compression and image quality. However, the information loss in the acquisition process sets the compression bounds. Further improvement in compression, without compromising the reconstruction quality is thus a challenge. In this work, we propose…
ETH Zurich（苏黎世联邦理工学院）；University Hospital（大学医院）；University of Zurich（苏黎世大学）；Uppsala University（乌普萨拉大学）
Fluorescence microscopy images contain several channels, each indicating a marker staining the sample. Since many different marker combinations are utilized in practice, it has been challenging to apply deep learning based segmentation models, which expect a predefined channel combination for all training samples as well as at inference for future application. Recent work circumvents this problem using a modality attention approach to be effective across any possible marker combination. However, for combinations that do not exist in a labeled training dataset, one cannot have any estimation of…
MTCurv: Deep learning for direct microtubule curvature mapping in noisy fluorescence microscopy images
TIAD Laboratory, Sciences and Technology Faculty, Sultan Moulay Slimane Univ.（TIAD实验室，科技与技术学院，苏丹穆莱·斯利曼大学）；CNRS, Univ. Rennes, IGDR (Institut de Génétique et Développement de Rennes), UMR6290（法国国家科学研究中心，雷恩大学，雷恩基因与发育研究所，UMR6290）
Washington University（华盛顿大学）；Washington University School of Medicine（华盛顿大学医学院）；PicoQuant GmbH（PicoQuant有限公司）
We have developed an automatic method for segmenting fluorescence lifetime (FLT) imaging microscopy (FLIM) images of cells inspired by a multi-resolution community detection (MCD) based network segmentation method. The image processing problem is framed as identifying segments with respective average FLTs against a background in FLIM images. The proposed method segments a FLIM image for a given resolution of the network composed using image pixels as the nodes and similarity between the pixels as the edges. In the resulting segmentation, low network resolution leads to larger segments and high…
There is a high demand for fully automated methods for the analysis of primary particle size distributions of agglomerates on transmission electron microscopy images. Therefore, a novel method, based on the utilization of artificial neural networks, was proposed, implemented and validated. The training of the artificial neural networks requires large quantities (up to several hundreds of thousands) of transmission electron microscopy images of agglomerates consisting of primary particles with known sizes. Since the manual evaluation of such large amounts of transmission electron microscopy ima…
A novel framework employing deep multi-attention channels network for the autonomous detection of metastasizing cells through fluorescence microscopy
Insigneo Institute for in-silico Medicine（Insigneo 计算机医学研究所）；University of Sheffield（谢菲尔德大学）；The Medical School, University of Sheffield（谢菲尔德大学医学院）；Madeira Chemistry Research Centre, University of Madeira（马德拉大学马德拉化学研究中心）；Karolinska Institutet（卡罗林斯卡学院）
A Zero-Shot Deep Image Prior Framework for Denoising and Deconvolution in Fluorescence Microscopy
Wenzhou Key Laboratory of Biomedical Imaging（温州生物医学成像重点实验室）；Zhejiang Key Laboratory of Soft Matter Biomedical Materials（浙江软物质生物医学材料重点实验室）；Wenzhou Institute, University of Chinese Academy of Sciences（中国科学院大学温州研究院）；Postgraduate Training Base Alliance of Wenzhou Medical University（温州医学院研究生联合培养基地联盟）
Semantic instance segmentation is the task of simultaneously partitioning an image into distinct segments while associating each pixel with a class label. In commonly used pipelines, segmentation and label assignment are solved separately since joint optimization is computationally expensive. We propose a greedy algorithm for joint graph partitioning and labeling derived from the efficient Mutex Watershed partitioning algorithm. It optimizes an objective function closely related to the Symmetric Multiway Cut objective and empirically shows efficient scaling behavior. Due to the algorithm's eff…
Due to image blurring image deconvolution is often used for studying biological structures in fluorescence microscopy. Fluorescence microscopy image volumes inherently suffer from intensity inhomogeneity, blur, and are corrupted by various types of noise which exacerbate image quality at deeper tissue depth. Therefore, quantitative analysis of fluorescence microscopy in deeper tissue still remains a challenge. This paper presents a three dimensional blind image deconvolution method for fluorescence microscopy using 3-way spatially constrained cycle-consistent adversarial networks. The restored…
Multi-Temporal Frames Projection for Dynamic Processes Fusion in Fluorescence Microscopy
Department of Computer Science, Swansea University（计算机科学系，萨瑟兰大学）；Department of Computational Perception, Johannes Kepler University（计算感知系，约翰尼斯·凯普勒大学）；Swansea University Medical School, Faculty of Medicine, Health（萨瑟兰大学医学院，医学与健康学院）
Purdue University（普渡大学）；Indiana University-Purdue University Indianapolis（印第安纳大学-普渡大学印第安纳波利斯分校）；Indiana University（印第安纳大学）
Fluorescence microscopy is an essential tool for the analysis of 3D subcellular structures in tissue. An important step in the characterization of tissue involves nuclei segmentation. In this paper, a two-stage method for segmentation of nuclei using convolutional neural networks (CNNs) is described. In particular, since creating labeled volumes manually for training purposes is not practical due to the size and complexity of the 3D data sets, the paper describes a method for generating synthetic microscopy volumes based on a spatially constrained cycle-consistent adversarial network. The prop…
De Montfort University（德蒙福特大学）；Instituto de Investigación Biomédica de Málaga – IBIMA（马拉加生物医学研究所）；Costa Rica Institute of Technology（哥斯达黎加理工学院）；University of Costa Rica（哥斯达黎加大学）；Rovira i Virgili University（罗维拉-威尔吉利大学）；University of Malaga（马拉加大学）
Cell instance segmentation in fluorescence microscopy images is becoming essential for cancer dynamics and prognosis. Data extracted from cancer dynamics allows to understand and accurately model different metabolic processes such as proliferation. This enables customized and more precise cancer treatments. However, accurate cell instance segmentation, necessary for further cell tracking and behavior analysis, is still challenging in scenarios with high cell concentration and overlapping edges. Within this framework, we propose a novel cell instance segmentation approach based on the well-know…
CyclePose -- Leveraging Cycle-Consistency for Annotation-Free Nuclei Segmentation in Fluorescence Microscopy
Friedrich-Alexander-Universität (FAU)（弗里德里希-亚历山大-埃尔朗根-纽伦堡大学（FAU））；Universität Würzburg（维尔茨堡大学）；Center for AI and Data Science (CAIDAS)（人工智能与数据科学中心（CAIDAS））
FluoCLIP: Stain-Aware Focus Quality Assessment in Fluorescence Microscopy
Division of AI and Software, Ewha Womans University（梨花女子大学人工智能与软件学部）；Bioimaging Data Curation Center (BDCC), Department of Life Science, Ewha Womans University（梨花女子大学生命科学系生物成像数据管理中心）；Analytical Solution Team, Daesang Corporation（大象集团分析解决方案团队）
ICube - UMR7357, CNRS（法国国家科学研究中心ICube实验室（UMR7357））；University of Strasbourg（斯特拉斯堡大学）；Centriole Lab（中心粒实验室）；Department of Molecular and Cellular Biology, University of Geneva（日内瓦大学分子与细胞生物学系）
Single particle reconstruction has recently emerged in 3D fluorescence microscopy as a powerful technique to improve the axial resolution and the degree of fluorescent labeling. It is based on the reconstruction of an average volume of a biological particle from the acquisition multiple views with unknown poses. Current methods are limited either by template bias, restriction to 2D data, high computational cost or a lack of robustness to low fluorescent labeling. In this work, we propose a single particle reconstruction method dedicated to convolutional models in 3D fluorescence microscopy tha…
Advances in fluorescence microscopy enable acquisition of 3D image volumes with better image quality and deeper penetration into tissue. Segmentation is a required step to characterize and analyze biological structures in the images and recent 3D segmentation using deep learning has achieved promising results. One issue is that deep learning techniques require a large set of groundtruth data which is impractical to annotate manually for large 3D microscopy volumes. This paper describes a 3D deep learning nuclei segmentation method using synthetic 3D volumes for training. A set of synthetic vol…
KAIST（韩国科学技术院）；Seoul National University College of Medicine（首尔大学医学院）；Neuroscience Research Institute, Seoul National University College of Medicine（首尔大学医学院神经科学研究所）；Pohang University of Science and Technology（浦项科技大学）
Volumetric imaging by fluorescence microscopy is often limited by anisotropic spatial resolution from inferior axial resolution compared to the lateral resolution. To address this problem, here we present a deep-learning-enabled unsupervised super-resolution technique that enhances anisotropic images in volumetric fluorescence microscopy. In contrast to the existing deep learning approaches that require matched high-resolution target volume images, our method greatly reduces the effort to put into practice as the training of a network requires as little as a single 3D image stack, without a pr…
Institute of Imaging and Computer Vision, RWTH Aachen University（亚琛工业大学成像与计算机视觉研究所）
Automated image processing approaches are indispensable for many biomedical experiments and help to cope with the increasing amount of microscopy image data in a fast and reproducible way. Especially state-of-the-art deep learning-based approaches most often require large amounts of annotated training data to produce accurate and generalist outputs, but they are often compromised by the general lack of those annotated data sets. In this work, we propose how conditional generative adversarial networks can be utilized to generate realistic image data for 3D fluorescence microscopy from annotatio…
Fluorescence microscopy is a widely used method among cell biologists for studying the localization and co-localization of fluorescent protein. For microbial cell biologists, these studies often include tedious and time-consuming manual segmentation of bacteria and of the fluorescence clusters or working with multiple programs. Here, we present MicroAnalyzer - a tool that automates these tasks by providing an end-to-end platform for microscope image analysis. While such tools do exist, they are costly, black-boxed programs. Microanalyzer offers an open-source alternative to these tools, allowi…
3D fluorescence microscopy of living organisms has increasingly become an essential and powerful tool in biomedical research and diagnosis. An exploding amount of imaging data has been collected, whereas efficient and effective computational tools to extract information from them are still lagging behind. This is largely due to the challenges in analyzing biological data. Interesting biological structures are not only small, but are often morphologically irregular and highly dynamic. Although tracking cells in live organisms has been studied for years, existing tracking methods for cells are n…
Volumetric imaging of samples using fluorescence microscopy plays an important role in various fields including physical, medical and life sciences. Here we report a deep learning-based volumetric image inference framework that uses 2D images that are sparsely captured by a standard wide-field fluorescence microscope at arbitrary axial positions within the sample volume. Through a recurrent convolutional neural network, which we term as Recurrent-MZ, 2D fluorescence information from a few axial planes within the sample is explicitly incorporated to digitally reconstruct the sample volume over …
European Laboratory for Non-Linear Spectroscopy (LENS), University of Florence（佛罗伦萨大学欧洲非线性光谱实验室）；Bioretics SrL（Bioretics SrL公司）；National Institute of Optics, National Research Council (INO-CNR)（国家研究委员会国家光学研究所）；Department of Physics（物理系）
Semantic segmentation of neuronal structures in 3D high-resolution fluorescence microscopy imaging of the human brain cortex can take advantage of bidimensional CNNs, which yield good results in neuron localization but lead to inaccurate surface reconstruction. 3D CNNs, on the other hand, would require manually annotated volumetric data on a large scale and hence considerable human effort. Semi-supervised alternative strategies which make use only of sparse annotations suffer from longer training times and achieved models tend to have increased capacity compared to 2D CNNs, needing more ground…
ETH Zurich（苏黎世联邦理工学院）；University Hospital and University of Zurich（苏黎世大学医院和苏黎世大学）；Uppsala University（乌普萨拉大学）
Fluorescence microscopy allows for a detailed inspection of cells, cellular networks, and anatomical landmarks by staining with a variety of carefully-selected markers visualized as color channels. Quantitative characterization of structures in acquired images often relies on automatic image analysis methods. Despite the success of deep learning methods in other vision applications, their potential for fluorescence image analysis remains underexploited. One reason lies in the considerable workload required to train accurate models, which are normally specific for a given combination of markers…
University of Notre Dame（圣母大学）；California Institute of Technology（加州理工学院）
Fluorescence microscopy has enabled a dramatic development in modern biology by visualizing biological organisms with micrometer scale resolution. However, due to the diffraction limit, sub-micron/nanometer features are difficult to resolve. While various super-resolution techniques are developed to achieve nanometer-scale resolution, they often either require expensive optical setup or specialized fluorophores. In recent years, deep learning has shown the potentials to reduce the technical barrier and obtain super-resolution from diffraction-limited images. For accurate results, conventional …
Kymographs are widely used to represent and anal- yse spatio-temporal dynamics of fluorescence markers along curvilinear biological compartments. These objects have a sin- gular geometry, thus kymograph reconstruction is inherently an analog image processing task. However, the existing approaches are essentially digital: the kymograph photometry is sampled directly from the time-lapse images. As a result, such kymographs rely on raw image data that suffer from the degradations entailed by the image formation process and the spatio-temporal resolution of the imaging setup. In this work, we addr…
Deblurring is a fundamental inverse problem in bioimaging. It requires modelling the point spread function (PSF), which captures the optical distortions entailed by the image formation process. The PSF limits the spatial resolution attainable for a given microscope. However, recent applications require a higher resolution, and have prompted the development of super-resolution techniques to achieve sub-pixel accuracy. This requirement restricts the class of suitable PSF models to analog ones. In addition, deblurring is computationally intensive, hence further requiring computationally efficient…
Department of Electrical Engineering, University of Notre Dame（圣母大学电气工程系）；Department of Biological Sciences, University of Notre Dame（圣母大学生物科学系）
Fluorescence microscopy has been a significant tool to observe long-term imaging of embryos (in vivo) growth over time. However, cumulative exposure is phototoxic to such sensitive live samples. While techniques like light-sheet fluorescence microscopy (LSFM) allow for reduced exposure, it is not well suited for deep imaging models. Other computational techniques are computationally expensive and often lack restoration quality. To address this challenge, one can use various low-dosage imaging techniques that are developed to achieve the 3D volume reconstruction using a few slices in the axial …
Fluorescence microscopy is a key driver to promote discoveries of biomedical research. However, with the limitation of microscope hardware and characteristics of the observed samples, the fluorescence microscopy images are susceptible to noise. Recently, a few self-supervised deep learning (DL) denoising methods have been proposed. However, the training efficiency and denoising performance of existing methods are relatively low in real scene noise removal. To address this issue, this paper proposed self-supervised image denoising method Noise2SR (N2SR) to train a simple and effective image den…
University of Notre Dame（圣母大学）；California Institute of Technology（加州理工学院）；Indiana University（印第安纳大学）
Fluorescence lifetime imaging microscopy (FLIM) systems are limited by their slow processing speed, low signal-to-noise ratio (SNR), and expensive and challenging hardware setups. In this work, we demonstrate applying a denoising convolutional network to improve FLIM SNR. The network will be integrated with an instant FLIM system with fast data acquisition based on analog signal processing, high SNR using high-efficiency pulse-modulation, and cost-effective implementation utilizing off-the-shelf radio-frequency components. Our instant FLIM system simultaneously provides the intensity, lifetime…
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Institute of Biophysics, Chinese Academy of Sciences（中国科学院生物物理研究所）；Hangzhou Dianzi University（杭州电子科技大学）
Super-resolution fluorescence microscopy, with a resolution beyond the diffraction limit of light, has become an indispensable tool to directly visualize biological structures in living cells at a nanometer-scale resolution. Despite advances in high-density super-resolution fluorescent techniques, existing methods still have bottlenecks, including extremely long execution time, artificial thinning and thickening of structures, and lack of ability to capture latent structures. Here we propose a novel deep learning guided Bayesian inference approach, DLBI, for the time-series analysis of high-de…
