Outdoor scene parsing models are often trained on ideal datasets and produce quality results. However, this leads to a discrepancy when applied to the real world. The quality of scene parsing, particularly sky classification, decreases in night time images, images involving varying weather conditions, and scene changes due to seasonal weather. This project focuses on approaching these challenges by using a state-of-the-art model in conjunction with a non-ideal dataset: SkyFinder and a subset from SUN database with Sky object. We focus specifically on sky segmentation, the task of determining sky and not-sky pixels, and improving upon an existing state-of-the-art model: RefineNet. As a result of our efforts, we have seen an improvement of 10-15% in the average MCR compared to the prior methods on SkyFinder dataset. We have also improved from an off-the shelf-model in terms of average mIOU by nearly 35%. Further, we analyze our trained models on images w.r.t two aspects: times of day and weather, and find that, in spite of facing same challenges as prior methods, our trained models significantly outperform them.
Latents of latents to delineate pixels: hybrid Matryoshka autoencoder-to-U-Net pairing for segmenting large medical images in GPU-poor and low-data regimes
School of Mathematics and Computer Science, Institute of Business Administration（商业管理学院数学与计算机科学学院）
Shandong University（山东大学）；City Public Passenger Transport Management Service Center of Jinan（济南市城市公共客运管理服务中心）
Most existing instance segmentation methods only focus on improving performance and are not suitable for real-time scenes such as autonomous driving. This paper proposes a real-time framework that segmenting and detecting 3D objects by depth. The framework is composed of two parallel branches: one for instance segmentation and another for object detection. We discretize the objects' depth into depth categories and transform the instance segmentation task into a pixel-level classification task. The Mask branch predicts pixel-level depth categories, and the 3D branch indicates instance-level dep…
Technical University Munich（慕尼黑工业大学）；Google Research（谷歌研究院）；RWTH Aachen University（亚琛工业大学）；MPI-IS（马克斯·普朗克智能系统研究所）；University of Tübingen（蒂宾根大学）
The task of assigning semantic classes and track identities to every pixel in a video is called video panoptic segmentation. Our work is the first that targets this task in a real-world setting requiring dense interpretation in both spatial and temporal domains. As the ground-truth for this task is difficult and expensive to obtain, existing datasets are either constructed synthetically or only sparsely annotated within short video clips. To overcome this, we introduce a new benchmark encompassing two datasets, KITTI-STEP, and MOTChallenge-STEP. The datasets contain long video sequences, provi…
Skip and Skip: Segmenting Medical Images with Prompts
Academy for Engineering and Technology, Fudan University（复旦大学工程与技术学院）；Engineering Research Center of AI and Robotics, Ministry of Education, China（教育部人工智能与机器人工程研究中心）；Jilin Provincial Key Laboratory of Intelligence Science（吉林省智能科学重点实验室）
Department of Radiology, School of Medicine, University of Pennsylvania（宾夕法尼亚大学医学院放射科）；School of Computer and Control Engineering, Yantai University（烟台大学计算机与控制工程学院）；The Children's Hospital of Philadelphia（费城儿童医院）
Purpose: To improve kidney segmentation in clinical ultrasound (US) images, we develop a new graph cuts based method to segment kidney US images by integrating original image intensity information and texture feature maps extracted using Gabor filters. Methods: To handle large appearance variation within kidney images and improve computational efficiency, we build a graph of image pixels close to kidney boundary instead of building a graph of the whole image. To make the kidney segmentation robust to weak boundaries, we adopt localized regional information to measure similarity between image p…
Ben-Gurion University（本-古里安大学）；Shamoon College of Engineering（沙蒙工程学院）
This paper publishes a natural and very complicated dataset of handwritten documents with multiply oriented and curved text lines, namely VML-MOC dataset. These text lines were written as remarks on the page margins by different writers over the years. They appear at different locations within the orientations that range between 0 and 180 or as curvilinear forms. We evaluate a multi-oriented Gaussian based method to segment these handwritten text lines that are skewed or curved in any orientation. It achieves a mean pixel Intersection over Union score of 80.96% on the test documents. The resul…
University of Washington（华盛顿大学）；Allen Institute for AI (AI2)（艾伦人工智能研究所 (AI2)）
Objects often occlude each other in scenes; Inferring their appearance beyond their visible parts plays an important role in scene understanding, depth estimation, object interaction and manipulation. In this paper, we study the challenging problem of completing the appearance of occluded objects. Doing so requires knowing which pixels to paint (segmenting the invisible parts of objects) and what color to paint them (generating the invisible parts). Our proposed novel solution, SeGAN, jointly optimizes for both segmentation and generation of the invisible parts of objects. Our experimental res…
Predicting the future is an important aspect for decision-making in robotics or autonomous driving systems, which heavily rely upon visual scene understanding. While prior work attempts to predict future video pixels, anticipate activities or forecast future scene semantic segments from segmentation of the preceding frames, methods that predict future semantic segmentation solely from the previous frame RGB data in a single end-to-end trainable model do not exist. In this paper, we propose a temporal encoder-decoder network architecture that encodes RGB frames from the past and decodes the fut…
We propose an end-to-end learning framework for segmenting generic objects in videos. Our method learns to combine appearance and motion information to produce pixel level segmentation masks for all prominent objects in videos. We formulate this task as a structured prediction problem and design a two-stream fully convolutional neural network which fuses together motion and appearance in a unified framework. Since large-scale video datasets with pixel level segmentations are problematic, we show how to bootstrap weakly annotated videos together with existing image recognition datasets for trai…
Due to the increasing availability of whole slide scanners facilitating digitization of histopathological tissue, there is a strong demand for the development of computer based image analysis systems. In this work, the focus is on the segmentation of the glomeruli constituting a highly relevant structure in renal histopathology, which has not been investigated before in combination with CNNs. We propose two different CNN cascades for segmentation applications with sparse objects. These approaches are applied to the problem of glomerulus segmentation and compared with conventional fully-convolu…
University of Copenhagen（哥本哈根大学）；Rigshospitalet（哥本哈根大学医院）
Tensor networks provide an efficient approximation of operations involving high dimensional tensors and have been extensively used in modelling quantum many-body systems. More recently, supervised learning has been attempted with tensor networks, primarily focused on tasks such as image classification. In this work, we propose a novel formulation of tensor networks for supervised image segmentation which allows them to operate on high resolution medical images. We use the matrix product state (MPS) tensor network on non-overlapping patches of a given input image to predict the segmentation mas…
University of Melbourne（墨尔本大学）；Monash University（莫纳什大学）；Cooperative Research Centre for Water Sensitive Cities（水敏感城市合作研究中心）；Arizona State University（亚利桑那州立大学）
Computer vision techniques enable automated detection of sky pixels in outdoor imagery. In urban climate, sky detection is an important first step in gathering information about urban morphology and sky view factors. However, obtaining accurate results remains challenging and becomes even more complex using imagery captured under a variety of lighting and weather conditions. To address this problem, we present a new sky pixel detection system demonstrated to produce accurate results using a wide range of outdoor imagery types. Images are processed using a selection of mean-shift segmentation, …
Isfahan University of Technology（伊斯法罕理工大学）；University of Michigan（密歇根大学）
Medical image analysis, especially segmenting a specific organ, has an important role in developing clinical decision support systems. In cardiac magnetic resonance (MR) imaging, segmenting the left and right ventricles helps physicians diagnose different heart abnormalities. There are challenges for this task, including the intensity and shape similarity between left ventricle and other organs, inaccurate boundaries and presence of noise in most of the images. In this paper we propose an automated method for segmenting the left ventricle in cardiac MR images. We first automatically extract th…
Despite the success of deep learning methods in medical image segmentation tasks, the human-level performance relies on massive training data with high-quality annotations, which are expensive and time-consuming to collect. The fact is that there exist low-quality annotations with label noise, which leads to suboptimal performance of learned models. Two prominent directions for segmentation learning with noisy labels include pixel-wise noise robust training and image-level noise robust training. In this work, we propose a novel framework to address segmenting with noisy labels by distilling ef…
Huazhong University of Science and Technology（华中科技大学）；Harvard University（哈佛大学）
Image segmentation is a fundamental vision task and a crucial step for many applications. In this paper, we propose a fast image segmentation method based on a novel super boundary-to-pixel direction (super-BPD) and a customized segmentation algorithm with super-BPD. Precisely, we define BPD on each pixel as a two-dimensional unit vector pointing from its nearest boundary to the pixel. In the BPD, nearby pixels from different regions have opposite directions departing from each other, and adjacent pixels in the same region have directions pointing to the other or each other (i.e., around media…
Medical Image and Signal Processing Research Center, Isfahan University of Medical Sciences（伊斯法罕医科大学医学图像与信号处理研究中心）；Snap Inc.（Snap公司）；ECE Department, Isfahan University of Technology（伊斯法罕理工大学电子与计算机工程系）；Iowa Institute for Biomedical Imaging, The University of Iowa（爱荷华大学生物医学成像研究所）
The novel corona-virus disease (COVID-19) pandemic has caused a major outbreak in more than 200 countries around the world, leading to a severe impact on the health and life of many people globally. As of mid-July 2020, more than 12 million people were infected, and more than 570,000 death were reported. Computed Tomography (CT) images can be used as an alternative to the time-consuming RT-PCR test, to detect COVID-19. In this work we propose a segmentation framework to detect chest regions in CT images, which are infected by COVID-19. We use an architecture similar to U-Net model, and train i…
This work examines the use of a fully convolutional net (FCN) to find an image segment, given a pixel within this segment region. The net receives an image, a point in the image and a region of interest (RoI ) mask. The net output is a binary mask of the segment in which the point is located. The region where the segment can be found is contained within the input RoI mask. Full image segmentation can be achieved by running this net sequentially, region-by-region on the image, and stitching the output segments into a single segmentation map. This simple method addresses two major challenges of …
Renmin University of China（中国人民大学）；Queen Mary University of London（伦敦玛丽女王大学）；Peking University（北京大学）
This paper presents a weakly supervised sparse learning approach to the problem of noisily tagged image parsing, or segmenting all the objects within a noisily tagged image and identifying their categories (i.e. tags). Different from the traditional image parsing that takes pixel-level labels as strong supervisory information, our noisily tagged image parsing is provided with noisy tags of all the images (i.e. image-level labels), which is a natural setting for social image collections (e.g. Flickr). By oversegmenting all the images into regions, we formulate noisily tagged image parsing as a …
The vast network of bridges in the United States raises a high requirement for maintenance and rehabilitation. The massive cost of manual visual inspection to assess bridge conditions is a burden to some extent. Advanced robots have been leveraged to automate inspection data collection. Automating the segmentations of multiclass elements and surface defects on the elements in the large volume of inspection image data would facilitate an efficient and effective assessment of the bridge condition. Training separate single-task networks for element parsing (i.e., semantic segmentation of multicla…
Research School of Computer Science（计算机科学研究院）；Australian National University（澳大利亚国立大学）
The problem of segmenting a given image into coherent regions is important in Computer Vision and many industrial applications require segmenting a known object into its components. Examples include identifying individual parts of a component for process control work in a manufacturing plant and identifying parts of a car from a photo for automatic damage detection. Unfortunately most of an object's parts of interest in such applications share the same pixel characteristics, having similar colour and texture. This makes segmenting the object into its components a non-trivial task for conventio…
In this paper we approach the novel problem of segmenting an image based on a natural language expression. This is different from traditional semantic segmentation over a predefined set of semantic classes, as e.g., the phrase "two men sitting on the right bench" requires segmenting only the two people on the right bench and no one standing or sitting on another bench. Previous approaches suitable for this task were limited to a fixed set of categories and/or rectangular regions. To produce pixelwise segmentation for the language expression, we propose an end-to-end trainable recurrent and con…
We propose an end-to-end learning framework for segmenting generic objects in both images and videos. Given a novel image or video, our approach produces a pixel-level mask for all "object-like" regions---even for object categories never seen during training. We formulate the task as a structured prediction problem of assigning an object/background label to each pixel, implemented using a deep fully convolutional network. When applied to a video, our model further incorporates a motion stream, and the network learns to combine both appearance and motion and attempts to extract all prominent ob…
A novel algorithm is proposed for segmenting an image into multiple levels using its mean and variance. Starting from the extreme pixel values at both ends of the histogram plot, the algorithm is applied recursively on sub-ranges computed from the previous step, so as to find a threshold level and a new sub-range for the next step, until no significant improvement in image quality can be achieved. The method makes use of the fact that a number of distributions tend towards Dirac delta function, peaking at the mean, in the limiting condition of vanishing variance. The procedure naturally provid…
University of Maryland, College Park（马里兰大学帕克分校）；University of North Texas（北德克萨斯大学）；DEVCOM Army Research Laboratory（DEVCOM陆军研究实验室）
The United States coastline spans 95,471 miles; a distance that cannot be effectively patrolled or secured by manual human effort alone. Unmanned Aerial Vehicles (UAVs) equipped with infrared cameras and deep-learning based algorithms represent a more efficient alternative for identifying and segmenting objects of interest - namely, ships. However, standard approaches to training these algorithms require large-scale datasets of densely labeled infrared maritime images. Such datasets are not publicly available and manually annotating every pixel in a large-scale dataset would have an extreme la…
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Cornell University（康奈尔大学）
We present a new framework for semantic segmentation without annotations via clustering. Off-the-shelf clustering methods are limited to curated, single-label, and object-centric images yet real-world data are dominantly uncurated, multi-label, and scene-centric. We extend clustering from images to pixels and assign separate cluster membership to different instances within each image. However, solely relying on pixel-wise feature similarity fails to learn high-level semantic concepts and overfits to low-level visual cues. We propose a method to incorporate geometric consistency as an inductive…
National Institute of Technology Karnataka（卡纳塔克邦国家理工学院）
In this paper, we have proposed a novel method for stereo disparity estimation by combining the existing methods of block based and region based stereo matching. Our method can generate dense disparity maps from disparity measurements of only 18% pixels of either the left or the right image of a stereo image pair. It works by segmenting the lightness values of image pixels using a fast implementation of K-Means clustering. It then refines those segment boundaries by morphological filtering and connected components analysis, thus removing a lot of redundant boundary pixels. This is followed by …
UC Berkeley（加州大学伯克利分校）；Universidad de los Andes, Colombia（哥伦比亚安第斯大学）；Microsoft Research（微软研究院）
The goal of this work is to replace objects in an RGB-D scene with corresponding 3D models from a library. We approach this problem by first detecting and segmenting object instances in the scene using the approach from Gupta et al. [13]. We use a convolutional neural network (CNN) to predict the pose of the object. This CNN is trained using pixel normals in images containing rendered synthetic objects. When tested on real data, it outperforms alternative algorithms trained on real data. We then use this coarse pose estimate along with the inferred pixel support to align a small number of prot…
The goal of our work is to perform pixel label semantic segmentation on 3D biomedical volumetric data. Manual annotation is always difficult for a large bio-medical dataset. So, we consider two cases where one dataset is fully labeled and the other dataset is assumed to be fully unlabelled. We first perform Semantic Segmentation on the fully labeled isotropic biomedical source data (FIBSEM) and try to incorporate the the trained model for segmenting the target unlabelled dataset(SNEMI3D)which shares some similarities with the source dataset in the context of different types of cellular bodies …
G-images refer to image data defined on irregular graph domains. This work elaborates a similarity-preserving Fuzzy C-Means (FCM) algorithm for G-image segmentation and aims to develop techniques and tools for segmenting G-images. To preserve the membership similarity between an arbitrary image pixel and its neighbors, a Kullback-Leibler divergence term on membership partition is introduced as a part of FCM. As a result, similarity-preserving FCM is developed by considering spatial information of image pixels for its robustness enhancement. Due to superior characteristics of a wavelet space, t…
Segmentation is the process of partitioning a digital image into multiple segments (sets of pixels). Such common segmentation tasks including segmenting written text or segmenting tumors from healthy brain tissue in an MRI image, etc. Chan-Vese model for active contours is a powerful and flexible method which is able to segment many types of images, including some that would be quite difficult to segment in means of "classical" segmentation - i.e., using thresholding or gradient based methods. This model is based on the Mumford-Shah functional for segmentation, and is used widely in the medica…
We introduce a new spectral method for image segmentation that incorporates long range relationships for global appearance modeling. The approach combines two different graphs, one is a sparse graph that captures spatial relationships between nearby pixels and another is a dense graph that captures pairwise similarity between all pairs of pixels. We extend the spectral method for Normalized Cuts to this setting by combining the transition matrices of Markov chains associated with each graph. We also derive an efficient method for sparsifying the dense graph of appearance relationships. This le…
Anomaly Segmentation for High-Resolution Remote Sensing Images Based on Pixel Descriptors
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（武汉大学测绘遥感信息工程国家重点实验室）；School of Remote Sensing and Information Engineering, Wuhan University（武汉大学遥感信息工程学院）
Carnegie Mellon University（卡内基梅隆大学）；New York University（纽约大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
We describe a system that builds a high dynamic-range and wide-angle image of the night sky by combining a large set of input images. The method makes use of pixel-rank information in the individual input images to improve a "consensus" pixel rank in the combined image. Because it only makes use of ranks and the complexity of the algorithm is linear in the number of images, the method is useful for large sets of uncalibrated images that might have undergone unknown non-linear tone mapping transformations for visualization or aesthetic reasons. We apply the method to images of the night sky (of…
Sun Yat-sen University Cancer Center（中山大学肿瘤防治中心）；Collaborative Innovation Center for Cancer Medicine（癌症医学协同创新中心）；State Key Laboratory of Oncology in South China（华南肿瘤学国家重点实验室）；Guangdong Key Laboratory of Nasopharyngeal Carcinoma Diagnosis and Therapy（广东省鼻咽癌诊治重点实验室）
Delineating the lesion area is an important task in image-based diagnosis. Pixel-wise classification is a popular approach to segmenting the region of interest. However, at fuzzy boundaries such methods usually result in glitches, discontinuity, or disconnection, inconsistent with the fact that lesions are solid and smooth. To overcome these undesirable artifacts, we propose the BezierSeg model which outputs bezier curves encompassing the region of interest. Directly modelling the contour with analytic equations ensures that the segmentation is connected, continuous, and the boundary is smooth…
Hamburg University of Technology（汉堡工业大学）；Universitätsklinikum Hamburg-Eppendorf（汉堡-埃彭多夫大学医学中心）
Colorectal Cancer(CRC) poses a great risk to public health. It is the third most common cause of cancer in the US. Development of colorectal polyps is one of the earliest signs of cancer. Early detection and resection of polyps can greatly increase survival rate to 90%. Manual inspection can cause misdetections because polyps vary in color, shape, size and appearance. To this end, Computer-Aided Diagnosis systems(CADx) has been proposed that detect polyps by processing the colonoscopic videos. The system acts a secondary check to help clinicians reduce misdetections so that polyps may be resec…
Faculty of Electrical Engineering, Computer Science and Information Technology（电气工程、计算机科学与信息技术学院）；Ghent University（根特大学）；Faculty of Engineering and Architecture（工程与建筑学院）；imec（比利时微电子研究中心）
Medical image segmentation often requires segmenting multiple elliptical objects on a single image. This includes, among other tasks, segmenting vessels such as the aorta in axial CTA slices. In this paper, we present a general approach to improving the semantic segmentation performance of neural networks in these tasks and validate our approach on the task of aorta segmentation. We use a cascade of two neural networks, where one performs a rough segmentation based on the U-Net architecture and the other performs the final segmentation on polar image transformations of the input. Connected com…
NCIS: Deep Color Gradient Maps Regression and Three-Class Pixel Classification for Enhanced Neuronal Cell Instance Segmentation in Nissl-Stained Histological Images
London South Bank University（伦敦南岸大学）；University of Padova（帕多瓦大学）
University of Cambridge（剑桥大学）；University of Bordeaux（波尔多大学）；CNRS（法国国家科学研究中心）；Unilever R&D Colworth（联合利华研发中心科尔沃斯）
The need for labour intensive pixel-wise annotation is a major limitation of many fully supervised learning methods for segmenting bioimages that can contain numerous object instances with thin separations. In this paper, we introduce a deep convolutional neural network for microscopy image segmentation. Annotation issues are circumvented by letting the network being trainable on coarse labels combined with only a very small number of images with pixel-wise annotations. We call this new labelling strategy `lazy' labels. Image segmentation is stratified into three connected tasks: rough inner r…
Fast Fourier Convolutional GAN for 30 m Clear-Sky Land Surface Temperature Gap-Free Reconstruction
Technical University of Munich（慕尼黑工业大学）；Helmholtz Centre for Polar and Marine Research, Alfred Wegener Institute（亥姆霍兹极地与海洋研究中心阿尔弗雷德·韦格纳研究所）；Swiss Federal Institute of Technology Zurich (ETH Zurich)（瑞士联邦理工学院苏黎世分校）
Unsupervised segmentation of large images using a Potts model Hamiltonian is unique in that segmentation is governed by a resolution parameter which scales the sensitivity to small clusters. Here, the input image is first modeled as a graph, which is then segmented by minimizing a Hamiltonian cost function defined on the graph and the respective segments. However, there exists no closed form solution of this optimization, and using previous iterative algorithmic solution techniques, the problem scales quadratically in the Input Length. Therefore, while Potts model segmentation gives accurate s…
Uni. of Bristol（布里斯托大学）；Uni. of Michigan（密歇根大学）；Uni. of Toronto（多伦多大学）
We introduce VISOR, a new dataset of pixel annotations and a benchmark suite for segmenting hands and active objects in egocentric video. VISOR annotates videos from EPIC-KITCHENS, which comes with a new set of challenges not encountered in current video segmentation datasets. Specifically, we need to ensure both short- and long-term consistency of pixel-level annotations as objects undergo transformative interactions, e.g. an onion is peeled, diced and cooked - where we aim to obtain accurate pixel-level annotations of the peel, onion pieces, chopping board, knife, pan, as well as the acting …
LenghuSky-8: An 8-Year All-Sky Cloud Dataset with Star-Aware Masks and Alt-Az Calibration for Segmentation and Nowcasting
State Key Laboratory of Dark Matter Physics, Tsung-Dao Lee Institute & School of Physics and Astronomy, Shanghai Jiao Tong University（暗物质物理国家重点实验室、李政道研究所及物理天文学院、上海交通大学）；National Astronomical Observatories, Chinese Academy of Sciences（中国科学院国家天文台）；School of Computer Technology and Application, Qinghai University（青海大学计算机技术与应用学院）
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Beijing University of Posts and Telecommunications（北京邮电大学）
Scarcity of annotated images hampers the building of automated solution for reliable COVID-19 diagnosis and evaluation from CT. To alleviate the burden of data annotation, we herein present a label-free approach for segmenting COVID-19 lesions in CT via pixel-level anomaly modeling that mines out the relevant knowledge from normal CT lung scans. Our modeling is inspired by the observation that the parts of tracheae and vessels, which lay in the high-intensity range where lesions belong to, exhibit strong patterns. To facilitate the learning of such patterns at a pixel level, we synthesize `les…
