University of Copenhagen（哥本哈根大学）；University of Amsterdam（阿姆斯特丹大学）；CIFAR（加拿大高级研究院）；University Hospital of Copenhagen（哥本哈根大学医院）；Erasmus MC, Rotterdam（鹿特丹伊拉斯姆斯医学中心）
Raghavendra Selvan, Thomas Kipf, Max Welling, Jesper H. Pedersen, Jens Petersen, Marleen de Bruijne
备注：
Extended Abstract submitted to MIDL, 2018. 3 pages
We present extraction of tree structures, such as airways, from image data as a graph refinement task. To this end, we propose a graph auto-encoder model that uses an encoder based on graph neural networks (GNNs) to learn embeddings from input node features and a decoder to predict connections between nodes. Performance of the GNN model is compared with mean-field networks in their ability to extract airways from 3D chest CT scans.
University of Copenhagen（哥本哈根大学）；University of Amsterdam（阿姆斯特丹大学）；Canadian Institute for Advanced Research（加拿大高级研究所）；University Hospital of Copenhagen（哥本哈根大学医院）；Erasmus Medical Center（伊拉斯姆斯医学中心）
We present tree extraction in 3D images as a graph refinement task, of obtaining a subgraph from an over-complete input graph. To this end, we formulate an approximate Bayesian inference framework on undirected graphs using mean field approximation (MFA). Mean field networks are used for inference based on the interpretation that iterations of MFA can be seen as feed-forward operations in a neural network. This allows us to learn the model parameters from training data using back-propagation algorithm. We demonstrate usefulness of the model to extract airway trees from 3D chest CT data. We fir…
University of Copenhagen（哥本哈根大学）；Rigshospitalet, University Hospital of Copenhagen（哥本哈根大学医院）；Erasmus MC Rotterdam（鹿特丹伊拉斯姆斯大学医学中心）
Knowledge of airway tree morphology has important clinical applications in diagnosis of chronic obstructive pulmonary disease. We present an automatic tree extraction method based on multiple hypothesis tracking and template matching for this purpose and evaluate its performance on chest CT images. The method is adapted from a semi-automatic method devised for vessel segmentation. Idealized tubular templates are constructed that match airway probability obtained from a trained classifier and ranked based on their relative significance. Several such regularly spaced templates form the local hyp…
Accurate, automatic and complete extraction of pulmonary airway in medical images plays an important role in analyzing thoracic CT volumes such as lung cancer detection, chronic obstructive pulmonary disease (COPD), and bronchoscopic-assisted surgery navigation. However, this task remains challenges, due to the complex tree-like structure of the airways. In this technical report, we use two-stage fully convolutional networks (FCNs) to automatically segment pulmonary airway in thoracic CT scans from multi-sites. Specifically, we firstly adopt a 3D FCN with U-shape network architecture to segmen…
We present a novel graph-based approach for labeling the anatomical branches of a given airway tree segmentation. The proposed method formulates airway labeling as a branch classification problem in the airway tree graph, where branch features are extracted using convolutional neural networks (CNN) and enriched using graph neural networks. Our graph neural network is structure-aware by having each node aggregate information from its local neighbors and position-aware by encoding node positions in the graph. We evaluated the proposed method on 220 airway trees from subjects with various severit…
Airway Segmentation Network for Enhanced Tubular Feature Extraction
School of Optical-electrical and Computer Engineering（光学电子与计算机工程学院）；University of Shanghai for Science and Technology（上海理工大学）；School of Information Technology（信息技术学院）；Jiangsu Open University（江苏开放大学）
Northeastern University（东北大学）；Shenyang University of Technology（沈阳工业大学）；The Chinese University of Hong Kong（香港中文大学）；Second Affiliated Hospital of Dalian Medical University（大连医科大学附属第二医院）；Central Hospital Affiliated to Shenyang Medical College（沈阳医学院附属中心医院）
Accurate airway extraction from computed tomography (CT) images is a critical step for planning navigation bronchoscopy and quantitative assessment of airway-related chronic obstructive pulmonary disease (COPD). The existing methods are challenging to sufficiently segment the airway, especially the high-generation airway, with the constraint of the limited label and cannot meet the clinical use in COPD. We propose a novel two-stage 3D contextual transformer-based U-Net for airway segmentation using CT images. The method consists of two stages, performing initial and refined airway segmentation…
Erasmus MC（伊拉斯谟大学医学中心）；Erasmus MC-Sophia Children Hospital（伊拉斯谟大学医学中心-索菲亚儿童医院）；University of Copenhagen（哥本哈根大学）
Segmentation of the airway tree from chest computed tomography (CT) images is critical for quantitative assessment of airway diseases including bronchiectasis and chronic obstructive pulmonary disease (COPD). However, obtaining an accurate segmentation of airways from CT scans is difficult due to the high complexity of airway structures. Recently, deep convolutional neural networks (CNNs) have become the state-of-the-art for many segmentation tasks, and in particular the so-called Unet architecture for biomedical images. However, its application to the segmentation of airways still remains a c…
Shanghai Jiao Tong University（上海交通大学）；Imperial College London（帝国理工学院）
Airway segmentation on CT scans is critical for pulmonary disease diagnosis and endobronchial navigation. Manual extraction of airway requires strenuous efforts due to the complicated structure and various appearance of airway. For automatic airway extraction, convolutional neural networks (CNNs) based methods have recently become the state-of-the-art approach. However, there still remains a challenge for CNNs to perceive the tree-like pattern and comprehend the connectivity of airway. To address this, we propose a voxel-connectivity aware approach named AirwayNet for accurate airway segmentat…
University of Copenhagen（哥本哈根大学）；Rigshospitalet, University Hospital of Copenhagen（哥本哈根大学医院）；Erasmus MC（伊拉斯谟医学中心）
Segmenting tree structures is common in several image processing applications. In medical image analysis, reliable segmentations of airways, vessels, neurons and other tree structures can enable important clinical applications. We present a framework for tracking tree structures comprising of elongated branches using probabilistic state-space models and Bayesian smoothing. Unlike most existing methods that proceed with sequential tracking of branches, we present an exploratory method, that is less sensitive to local anomalies in the data due to acquisition noise and/or interfering structures. …
RepAir: A Framework for Airway Segmentation and Discontinuity Correction in CT
University of Michigan（密歇根大学）；D Medical, Inc.（4D医疗公司）；University of Massachusetts（马萨诸塞大学）；Emory University（埃默里大学）；Georgia Institute of Technology（佐治亚理工学院）
University of Amsterdam（阿姆斯特丹大学）；Canadian Institute of Advanced Research（加拿大高等研究院）；Erasmus MC（伊拉斯姆斯医学中心）；Rigshospitalet（哥本哈根大学医院）；University of Copenhagen（哥本哈根大学）
Graph refinement, or the task of obtaining subgraphs of interest from over-complete graphs, can have many varied applications. In this work, we extract trees or collection of sub-trees from image data by, first deriving a graph-based representation of the volumetric data and then, posing the tree extraction as a graph refinement task. We present two methods to perform graph refinement. First, we use mean-field approximation (MFA) to approximate the posterior density over the subgraphs from which the optimal subgraph of interest can be estimated. Mean field networks (MFNs) are used for inferenc…
In this research project, we put forward an advanced method for airway segmentation based on the existent convolutional neural network (CNN) and graph neural network (GNN). The method is originated from the vessel segmentation, but we ameliorate it and enable the novel model to perform better for datasets from computed tomography (CT) scans. Current methods for airway segmentation are considering the regular grid only. No matter what the detailed model is, including the 3-dimensional CNN or 2-dimensional CNN in three directions, the overall graph structures are not taken into consideration. In…
Erasmus MC（伊拉斯姆斯医学中心）；Erasmus MC-Sophia Children Hospital（伊拉斯姆斯医学中心索菲亚儿童医院）；University of Copenhagen（哥本哈根大学）；Copenhagen University Hospital（哥本哈根大学医院）；Herlev-Gentofte Hospital（海莱乌-根措夫特医院）
This paper presents a fully automatic and end-to-end optimised airway segmentation method for thoracic computed tomography, based on the U-Net architecture. We use a simple and low-memory 3D U-Net as backbone, which allows the method to process large 3D image patches, often comprising full lungs, in a single pass through the network. This makes the method simple, robust and efficient. We validated the proposed method on three datasets with very different characteristics and various airway abnormalities: i) a dataset of pediatric patients including subjects with cystic fibrosis, ii) a subset of…
National Heart and Lung Institute, Imperial College London（帝国理工学院国家心肺研究所）；University of the Basque Country UPV/EHU（巴斯克大学（UPV/EHU））；TECNALIA, Basque Research and Technology Alliance (BRTA)（巴斯克研究与技术联盟（BRTA）泰克纳利亚研究中心）；Technical University of Munich（慕尼黑工业大学）；King Abdulaziz University（阿卜杜勒阿齐兹国王大学）；University of Alberta（阿尔伯塔大学）；The Systems Research Institute, Polish Academy of Sciences（波兰科学院系统研究所）；Royal Brompton Hospital（皇家布朗普顿医院）
Airway segmentation is crucial for the examination, diagnosis, and prognosis of lung diseases, while its manual delineation is unduly burdensome. To alleviate this time-consuming and potentially subjective manual procedure, researchers have proposed methods to automatically segment airways from computerized tomography (CT) images. However, some small-sized airway branches (e.g., bronchus and terminal bronchioles) significantly aggravate the difficulty of automatic segmentation by machine learning models. In particular, the variance of voxel values and the severe data imbalance in airway branch…
Erasmus MC（伊拉斯姆斯医学中心）；Erasmus MC-Sophia Children Hospital（伊拉斯姆斯医学中心索菲亚儿童医院）；University of Copenhagen（哥本哈根大学）；Herlev-Gentofte Hospital, Copenhagen University Hospital（哥本哈根大学医院海尔鲁夫-根措夫特医院）
We present an end-to-end deep learning segmentation method by combining a 3D UNet architecture with a graph neural network (GNN) model. In this approach, the convolutional layers at the deepest level of the UNet are replaced by a GNN-based module with a series of graph convolutions. The dense feature maps at this level are transformed into a graph input to the GNN module. The incorporation of graph convolutions in the UNet provides nodes in the graph with information that is based on node connectivity, in addition to the local features learnt through the downsampled paths. This information can…
Institute of Medical Robotics, Shanghai Jiao Tong University（上海交通大学医学机器人研究所）；School of Computer Science and Technology, Guizhou University（贵州大学计算机科学与技术学院）；Shanghai Chest Hospital, Shanghai Jiao Tong University（上海交通大学附属胸科医院）；Université de Lyon, INSA Lyon, CREATIS, CNRS, INSERM（里昂大学、里昂国立应用科学学院、CREATIS实验室、法国国家科学研究中心、法国国家健康与医学研究院）
Training convolutional neural networks (CNNs) for segmentation of pulmonary airway, artery, and vein is challenging due to sparse supervisory signals caused by the severe class imbalance between tubular targets and background. We present a CNNs-based method for accurate airway and artery-vein segmentation in non-contrast computed tomography. It enjoys superior sensitivity to tenuous peripheral bronchioles, arterioles, and venules. The method first uses a feature recalibration module to make the best use of features learned from the neural networks. Spatial information of features is properly i…
School of Mechanical engineering and Automation, Fuzhou University（福州大学机械工程及自动化学院）；School of Physics and Information Engineering, Fuzhou University（福州大学物理与信息工程学院）；Thoracic Department, Fujian Medical University Union Hospital（福建医科大学附属协和医院胸外科）
Automatic airway segmentation from chest computed tomography (CT) scans plays an important role in pulmonary disease diagnosis and computer-assisted therapy. However, low contrast at peripheral branches and complex tree-like structures remain as two mainly challenges for airway segmentation. Recent research has illustrated that deep learning methods perform well in segmentation tasks. Motivated by these works, a coarse-to-fine segmentation framework is proposed to obtain a complete airway tree. Our framework segments the overall airway and small branches via the multi-information fusion convol…
Institute of Medical Robotics, Shanghai Jiao Tong University（上海交通大学医疗机器人研究院）；Medical Image Centre, The First Affiliated Hospital of Jinan University（暨南大学附属第一医院医学影像中心）；Department of Computer Science and Software Engineering, Swinburne University of Technology（斯威本科技大学计算机科学与软件工程系）；Shanghai Center for Brain Science and Brain-Inspired Technology（上海脑科学与类脑研究中心）
3D Convolutional Neural Networks (CNNs) have been widely adopted for airway segmentation. The performance of 3D CNNs is greatly influenced by the dataset while the public airway datasets are mainly clean CT scans with coarse annotation, thus difficult to be generalized to noisy CT scans (e.g. COVID-19 CT scans). In this work, we proposed a new dual-stream network to address the variability between the clean domain and noisy domain, which utilizes the clean CT scans and a small amount of labeled noisy CT scans for airway segmentation. We designed two different encoders to extract the transferab…
The Plug-and-Play (PnP) framework was recently introduced for low-dose CT reconstruction to leverage the interpretability and the flexibility of model-based methods to incorporate various plugins, such as trained deep learning (DL) neural networks. However, the benefits of PnP vs. state-of-the-art DL methods have not been clearly demonstrated. In this work, we proposed an improved PnP framework to address the previous limitations and develop clinical-relevant segmentation metrics for quantitative result assessment. Compared with the DL alone methods, our proposed PnP framework was slightly inf…
Validated respiratory drug deposition predictions from 2D and 3D medical images with statistical shape models and convolutional neural networks
Heriot-Watt University（赫瑞瓦特大学）；Royal Hospital for Children and Young People, NHS Lothian（NHS洛锡安区儿童与青少年皇家医院）；University Hospital Southampton NHS Foundation Trust（南安普顿大学医院NHS基金会信托）；Brunel University（布鲁内尔大学）；University of Edinburgh（爱丁堡大学）
Automatic pathological pulmonary lobe segmentation(PPLS) enables regional analyses of lung disease, a clinically important capability. Due to often incomplete lobe boundaries, PPLS is difficult even for experts, and most prior art requires inference from contextual information. To address this, we propose a novel PPLS method that couples deep learning with the random walker (RW) algorithm. We first employ the recent progressive holistically-nested network (P-HNN) model to identify potential lobar boundaries, then generate final segmentations using a RW that is seeded and weighted by the P-HNN …
National University of Science and Technology (MISiS)（国立研究型技术大学（莫斯科钢铁合金学院））
Despite the highest classification accuracy in wide varieties of application areas, artificial neural network has one disadvantage. The way this Network comes to a decision is not easily comprehensible. The lack of explanation ability reduces the acceptability of neural network in data mining and decision system. This drawback is the reason why researchers have proposed many rule extraction algorithms to solve the problem. Recently, Deep Neural Network (DNN) is achieving a profound result over the standard neural network for classification and recognition problems. It is a hot machine learning…
One of the most arduous and captivating domains under image processing is handwritten character recognition. In this paper we have proposed a feature extraction technique which is a combination of unique features of geometric, zone-based hybrid, gradient features extraction approaches and three different neural networks namely the Multilayer Perceptron network using Backpropagation algorithm (MLP BP), the Multilayer Perceptron network using Levenberg-Marquardt algorithm (MLP LM) and the Convolutional neural network (CNN) which have been implemented along with the Minimum Distance Classifier (M…
Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University（上海交通大学图像处理与模式识别研究所）；Institute of Medical Robotics, Shanghai Jiao Tong University（上海交通大学医疗机器人研究院）；Department of Respiratory and Critical Care Medicine, Department of Respiratory Endoscopy, Shanghai Chest Hospital, Shanghai Engineering Research Center of Respiratory Endoscopy（上海市胸科医院呼吸与危重症医学科、呼吸内镜中心、上海市呼吸内镜工程技术研究中心）
Airway segmentation is critical for virtual bronchoscopy and computer-aided pulmonary disease analysis. In recent years, convolutional neural networks (CNNs) have been widely used to delineate the bronchial tree. However, the segmentation results of the CNN-based methods usually include many discontinuous branches, which need manual repair in clinical use. A major reason for the breakages is that the appearance of the airway wall can be affected by the lung disease as well as the adjacency of the vessels, while the network tends to overfit to these special patterns in the training set. To lear…
Detecting and extracting information from Machine-Readable Zone (MRZ) on passports and visas is becoming increasingly important for verifying document authenticity. However, computer vision methods for performing similar tasks, such as optical character recognition (OCR), fail to extract the MRZ given digital images of passports with reasonable accuracy. We present a specially designed model based on convolutional neural networks that is able to successfully extract MRZ information from digital images of passports of arbitrary orientation and size. Our model achieved 100% MRZ detection rate an…
In this paper, an approach to the problem of automatic facial feature extraction from a still frontal posed image and classification and recognition of facial expression and hence emotion and mood of a person is presented. Feed forward back propagation neural network is used as a classifier for classifying the expressions of supplied face into seven basic categories like surprise, neutral, sad, disgust, fear, happy and angry. For face portion segmentation and localization, morphological image processing operations are used. Permanent facial features like eyebrows, eyes, mouth and nose are extr…
Université Pierre et Marie Curie（皮埃尔和玛丽·居里大学）；SIGnal processing and MAchine learning Lab, ESPCI ParisTech, PSL Research University（巴黎高等物理化工学院信号处理与机器学习实验室，巴黎文理研究大学）；Vocal Tract Visualization Lab, University of Maryland Dental School（马里兰大学牙科学院声道可视化实验室）
Studying tongue motion during speech using ultrasound is a standard procedure, but automatic ultrasound image labelling remains a challenge, as standard tongue shape extraction methods typically require human intervention. This article presents a method based on deep neural networks to automatically extract tongue contour from ultrasound images on a speech dataset. We use a deep autoencoder trained to learn the relationship between an image and its related contour, so that the model is able to automatically reconstruct contours from the ultrasound image alone. In this paper, we use an automati…
Tables are widely used in several types of documents since they can bring important information in a structured way. In scientific papers, tables can sum up novel discoveries and summarize experimental results, making the research comparable and easily understandable by scholars. Several methods perform table analysis working on document images, losing useful information during the conversion from the PDF files since OCR tools can be prone to recognition errors, in particular for text inside tables. The main contribution of this work is to tackle the problem of table extraction, exploiting Gra…
National Institute of Technology Karnataka（卡纳塔克国立理工学院）
The proposed framework in this paper has the primary objective of classifying the facial expression shown by a person. These classifiable expressions can be any one of the six universal emotions along with the neutral emotion. After the initial facial localization is performed, facial landmark detection and feature extraction are applied where in the landmarks are determined to be the fiducial features: the eyebrows, eyes, nose and lips. This is primarily done using state-of-the-art facial landmark detection algorithms as well as traditional edge and corner point detection methods using Sobel …
Graz University of Technology（格拉茨工业大学）；VRVis（VRVis视觉计算研究中心）
While most state-of-the-art instance segmentation methods produce binary segmentation masks, geographic and cartographic applications typically require precise vector polygons of extracted objects instead of rasterized output. This paper introduces PolyWorld, a neural network that directly extracts building vertices from an image and connects them correctly to create precise polygons. The model predicts the connection strength between each pair of vertices using a graph neural network and estimates the assignments by solving a differentiable optimal transport problem. Moreover, the vertex posi…
School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）；Tencent America（腾讯美国）；Advanced Institute of Information Technology, Peking University（北京大学高等信息科学技术研究院）
We deal with the controllable person image synthesis task which aims to re-render a human from a reference image with explicit control over body pose and appearance. Observing that person images are highly structured, we propose to generate desired images by extracting and distributing semantic entities of reference images. To achieve this goal, a neural texture extraction and distribution operation based on double attention is described. This operation first extracts semantic neural textures from reference feature maps. Then, it distributes the extracted neural textures according to the spati…
Deep Convolutional Neural Networks with Zero-Padding: Feature Extraction and Learning
Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences（中国科学院机器人与智能制造创新研究院）；Xi’an Jiaotong University（西安交通大学）；University of Sydney（悉尼大学）
Moscow Institute of Physics and Technology（莫斯科物理技术学院）；National Research University Higher School of Economics（国家研究型高等经济大学）；Universität Tübingen（图宾根大学）；Ecole Polytechnique（巴黎综合理工学院）
Astrophysical images in the GeV band are challenging to analyze due to the strong contribution of the background and foreground astrophysical diffuse emission and relatively broad point spread function of modern space-based instruments. In certain cases, even finding of point sources on the image becomes a non-trivial task. We present a method for point sources extraction using a convolution neural network (CNN) trained on our own artificial data set which imitates images from the Fermi Large Area Telescope. These images are raw count photon maps of 10x10 degrees covering energies from 1 to 10…
Huazhong University of Science and Technology（华中科技大学）；Yichang Testing Technique Research Institute（宜昌测试技术研究所）；Changsha University（长沙学院）
Convolutional neural networks (CNNs) have achieved superior performance but still lack clarity about the nature and properties of feature extraction. In this paper, by analyzing the sensitivity of neural networks to frequencies and scales, we find that neural networks not only have low- and medium-frequency biases but also prefer different frequency bands for different classes, and the scale of objects influences the preferred frequency bands. These observations lead to the hypothesis that neural networks must learn the ability to extract features at various scales and frequencies. To corrobor…
Many earth observation programs such as Landsat, Sentinel, SPOT, and Pleiades produce huge volume of medium to high resolution multi-spectral images every day that can be organized in time series. In this work, we exploit both temporal and spatial information provided by these images to generate land cover maps. For this purpose, we combine a fully convolutional neural network with a convolutional long short-term memory. Implementation details of the proposed spatio-temporal neural network architecture are provided. Experimental results show that the temporal information provided by time serie…
Automatic data extraction from charts is challenging for two reasons: there exist many relations among objects in a chart, which is not a common consideration in general computer vision problems; and different types of charts may not be processed by the same model. To address these problems, we propose a framework of a single deep neural network, which consists of object detection, text recognition and object matching modules. The framework handles both bar and pie charts, and it may also be extended to other types of charts by slight revisions and by augmenting the training data. Our model pe…
Optical Coherence Tomography allows ophthalmologist to obtain cross-section imaging of eye retina. Assisted with digital image analysis methods, effective disease detection could be performed. Various methods exist to extract feature from OCT images. The proposed study aims to compare the effectiveness of handcrafted and deep neural network features. The evaluated dataset consist of 32339 instances distributed in four classes, namely CNV, DME, DRUSEN, and NORMAL. The methods are Histogram of Oriented Gradient (HOG), Local Binary Pattern (LBP), DenseNet-169, and ResNet50. As a result, the deep …
American International University-Bangladesh（孟加拉国国际大学）
Artificial Neural Network is among the most popular algorithm for supervised learning. However, Neural Networks have a well-known drawback of being a "Black Box" learner that is not comprehensible to the Users. This lack of transparency makes it unsuitable for many high risk tasks such as medical diagnosis that requires a rational justification for making a decision. Rule Extraction methods attempt to curb this limitation by extracting comprehensible rules from a trained Network. Many such extraction algorithms have been developed over the years with their respective strengths and weaknesses. …
The improvements in spectral and spatial resolution of the satellite images have facilitated the automatic extraction and identification of the features from satellite images and aerial photographs. An automatic object extraction method is presented for extracting and identifying the various objects from satellite images and the accuracy of the system is verified with regard to IRS satellite images. The system is based on neural network and simulates the process of visual interpretation from remote sensing images and hence increases the efficiency of image analysis. This approach obtains the b…
