Comprehensive Saliency Fusion for Object Co-segmentation
Indraprastha Institute of Information Technology Delhi(印度德里英迪拉普拉萨塔信息技术学院)
Harshit Singh Chhabra, Koteswar Rao Jerripothula
备注:
Published in IEEE ISM 2021. Please cite this paper in the following manner. H. S. Chhabra and K. Rao Jerripothula, "Comprehensive Saliency Fusion for Object Co-segmentation," 2021 IEEE International Symposium on Multimedia (ISM), 2021, pp. 107-110, doi: 10.1109/ISM52913.2021.00026
发表信息:
2021 IEEE International Symposium on Multimedia (ISM), 2021, pp. 107-110
Object co-segmentation has drawn significant attention in recent years, thanks to its clarity on the expected foreground, the shared object in a group of images. Saliency fusion has been one of the promising ways to carry it out. However, prior works either fuse saliency maps of the same image or saliency maps of different images to extract the expected foregrounds. Also, they rely on hand-crafted saliency extraction and correspondence processes in most cases. This paper revisits the problem and proposes fusing saliency maps of both the same image and different images. It also leverages advances in deep learning for the saliency extraction and correspondence processes. Hence, we call it comprehensive saliency fusion. Our experiments reveal that our approach achieves much-improved object co-segmentation results compared to prior works on important benchmark datasets such as iCoseg, MSRC, and Internet Images.
National Research Council, Institute for the Application of Calculus(国家研究委员会,计算应用研究所)
Skin lesion segmentation is one of the crucial steps for an efficient non-invasive computer-aided early diagnosis of melanoma. This paper investigates how color information, besides saliency, can be used to determine the pigmented lesion region automatically. Unlike most existing segmentation methods using only the saliency in order to discriminate against the skin lesion from the surrounding regions, we propose a novel method employing a binarization process coupled with new perceptual criteria, inspired by the human visual perception, related to the properties of saliency and color of the in…
Existing weakly supervised semantic segmentation (WSSS) methods usually utilize the results of pre-trained saliency detection (SD) models without explicitly modeling the connections between the two tasks, which is not the most efficient configuration. Here we propose a unified multi-task learning framework to jointly solve WSSS and SD using a single network, \ie saliency, and segmentation network (SSNet). SSNet consists of a segmentation network (SN) and a saliency aggregation module (SAM). For an input image, SN generates the segmentation result and, SAM predicts the saliency of each category…
Saliency map detection, as a method for detecting important regions of an image, is used in many applications such as image classification and recognition. We propose that context detection could have an essential role in image saliency detection. This requires extraction of high level features. In this paper a saliency map is proposed, based on image context detection using semantic segmentation as a high level feature. Saliency map from semantic information is fused with color and contrast based saliency maps. The final saliency map is then generated. Simulation results for Pascal-voc11 imag…
Weakly-supervised image segmentation is an important task in computer vision. A key problem is how to obtain high quality objects location from image-level category. Classification activation mapping is a common method which can be used to generate high-precise object location cues. However these location cues are generally very sparse and small such that they can not provide effective information for image segmentation. In this paper, we propose a saliency guided image segmentation network to resolve this problem. We employ a self-attention saliency method to generate subtle saliency maps, an…
We propose a new saliency-guided method for generating supervoxels in 3D space. Rather than using an evenly distributed spatial seeding procedure, our method uses visual saliency to guide the process of supervoxel generation. This results in densely distributed, small, and precise supervoxels in salient regions which often contain objects, and larger supervoxels in less salient regions that often correspond to background. Our approach largely improves the quality of the resulting supervoxel segmentation in terms of boundary recall and under-segmentation error on publicly available benchmarks.
Department of CS & IT, SUIT Peshawar(白沙瓦SUIT计算机科学与信息技术系)
Skin lesion segmentation can be rather a challenging task owing to the presence of artifacts, low contrast between lesion and boundary, color variegation, fuzzy skin lesion borders and heterogeneous background in dermoscopy images. In this paper, we propose a simple yet effective saliency-based approach derived in the frequency and spatial domain to detect pigmented skin lesion. Two color models are utilized for the construction of these maps. We suggest a different metric for each color model to design map in the spatial domain via color features. The map in the frequency domain is generated …
University of British Columbia(不列颠哥伦比亚大学);TELUS Communications Incorporation(TELUS通信公司)
Saliency prediction for Standard Dynamic Range (SDR) videos has been well explored in the last decade. However, limited studies are available on High Dynamic Range (HDR) Visual Attention Models (VAMs). Considering that the characteristic of HDR content in terms of dynamic range and color gamut is quite different than those of SDR content, it is essential to identify the importance of different saliency attributes of HDR videos for designing a VAM and understand how to combine these features. To this end we propose a learning-based visual saliency fusion method for HDR content (LVBS-HDR) to com…
Predicting saliency in videos is a challenging problem due to complex modeling of interactions between spatial and temporal information, especially when ever-changing, dynamic nature of videos is considered. Recently, researchers have proposed large-scale datasets and models that take advantage of deep learning as a way to understand what's important for video saliency. These approaches, however, learn to combine spatial and temporal features in a static manner and do not adapt themselves much to the changes in the video content. In this paper, we introduce Gated Fusion Network for dynamic sal…
Saliency computation has become a popular research field for many applications due to the useful information provided by saliency maps. For a saliency map, local relations around the salient regions in multi-channel perspective should be taken into consideration by aiming uniformity on the region of interest as an internal approach. And, irrelevant salient regions have to be avoided as much as possible. Most of the works achieve these criteria with external processing modules; however, these can be accomplished during the conspicuity map fusion process. Therefore, in this paper, a new model is…
Active contour models have achieved prominent success in the area of image segmentation, allowing complex objects to be segmented from the background for further analysis. Existing models can be divided into region-based active contour models and edge-based active contour models. However, both models use direct image data to achieve segmentation and face many challenging problems in terms of the initial contour position, noise sensitivity, local minima and inefficiency owing to the in-homogeneity of image intensities. The saliency map of an image changes the image representation, making it mor…
In this paper, we present a fast yet effective method for pixel-level scale-invariant image fusion in spatial domain based on the scale-space theory. Specifically, we propose a scale-invariant structure saliency selection scheme based on the difference-of-Gaussian (DoG) pyramid of images to build the weights or activity map. Due to the scale-invariant structure saliency selection, our method can keep both details of small size objects and the integrity information of large size objects in images. In addition, our method is very efficient since there are no complex operation involved and easy t…
Instance-level quantitative saliency in multiple sclerosis lesion segmentation
Translational Imaging in Neurology (ThINk)(神经学转化成像(ThINk));Department of Medicine and Biomedical Engineering, University Hospital Basel and University of Basel(医学与生物医学工程系,巴塞尔大学医院和巴塞尔大学);Department of Neurology, University Hospital Basel(神经病学系,巴塞尔大学医院);Research Center for Clinical Neuroimmunology and Neuroscience Basel (RC2NB)(巴塞尔临床神经免疫学和神经科学研究中心(RC2NB));MedGIFT, Institute of Informatics, School of Management, HES-SO Valais-Wallis University of Applied Sciences and Arts Western Switzerland(MedGIFT,信息学院,管理学院,瓦莱-迪-瓦尔省应用科学与艺术西方瑞士大学);CIBM Center for Biomedical Imaging, Lausanne, Switzerland(洛桑瑞士生物医学成像中心);Radiology Department, Lausanne University Hospital (CHUV) and University of Lausanne(放射科,洛桑大学医院(CHUV)和洛桑大学)
Attention-Guided Lidar Segmentation and Odometry Using Image-to-Point Cloud Saliency Transfer
University of Tsukuba(筑波大学);National Institute of Advanced Industrial Science and Technology(日本产业技术综合研究所);CNRS-AIST Joint Robotics Laboratory (JRL)(法国国家科学研究中心-日本产业技术综合研究所联合机器人实验室)
Tarbiat Modares University(塔比亚特·莫达雷斯大学);NRP company(NRP公司);University of Sheffield(谢菲尔德大学)
Lesion segmentation is the first step in most automatic melanoma recognition systems. Deficiencies and difficulties in dermoscopic images such as color inconstancy, hair occlusion, dark corners and color charts make lesion segmentation an intricate task. In order to detect the lesion in the presence of these problems, we propose a supervised saliency detection method tailored for dermoscopic images based on the discriminative regional feature integration (DRFI). DRFI method incorporates multi-level segmentation, regional contrast, property, background descriptors, and a random forest regressor…
In this paper, we propose several novel deep learning methods for object saliency detection based on the powerful convolutional neural networks. In our approach, we use a gradient descent method to iteratively modify an input image based on the pixel-wise gradients to reduce a cost function measuring the class-specific objectness of the image. The pixel-wise gradients can be efficiently computed using the back-propagation algorithm. The discrepancy between the modified image and the original one may be used as a saliency map for the image. Moreover, we have further proposed several new trainin…
We propose an image super resolution(ISR) method using generative adversarial networks (GANs) that takes a low resolution input fundus image and generates a high resolution super resolved (SR) image upto scaling factor of $16$. This facilitates more accurate automated image analysis, especially for small or blurred landmarks and pathologies. Local saliency maps, which define each pixel's importance, are used to define a novel saliency loss in the GAN cost function. Experimental results show the resulting SR images have perceptual quality very close to the original images and perform better tha…
Visualisation, Faculty of Computer Science, Technische Universit\" a t Dresden, N\" o thnitzer Stra e 46, 01187 Dresden, Germany
Image fusion helps in merging two or more images to construct a more informative single fused image. Recently, unsupervised learning based convolutional neural networks (CNN) have been utilized for different types of image fusion tasks such as medical image fusion, infrared-visible image fusion for autonomous driving as well as multi-focus and multi-exposure image fusion for satellite imagery. However, it is challenging to analyze the reliability of these CNNs for the image fusion tasks since no groundtruth is available. This led to the use of a wide variety of model architectures and optimiza…
High dynamic range (HDR) imaging enables to immortalize natural scenes similar to the way that they are perceived by human observers. With regular low dynamic range (LDR) capture/display devices, significant details may not be preserved in images due to the huge dynamic range of natural scenes. To minimize the information loss and produce high quality HDR-like images for LDR screens, this study proposes an efficient multi-exposure fusion (MEF) approach with a simple yet effective weight extraction method relying on principal component analysis, adaptive well-exposedness and saliency maps. Thes…
In this paper we propose two saliency models for salient object segmentation based on a hierarchical image segmentation, a tree-like structure that represents regions at different scales from the details to the whole image (e.g. gPb-UCM, BPT). The first model is based on a hierarchy of image partitions. The saliency at each level is computed on a region basis, taking into account the contrast between regions. The maps obtained for the different partitions are then integrated into a final saliency map. The second model directly works on the structure created by the segmentation algorithm, compu…
Efficiently exploiting multi-modal inputs for accurate RGB-D saliency detection is a topic of high interest. Most existing works leverage cross-modal interactions to fuse the two streams of RGB-D for intermediate features' enhancement. In this process, a practical aspect of the low quality of the available depths has not been fully considered yet. In this work, we aim for RGB-D saliency detection that is robust to the low-quality depths which primarily appear in two forms: inaccuracy due to noise and the misalignment to RGB. To this end, we propose a robust RGB-D fusion method that benefits fr…
College of Information Science & Electronic Engineering, Zhejiang University(浙江大学信息科学与电子工程学院)
RGB-D salient object detection aims to identify the most visually distinctive objects in a pair of color and depth images. Based upon an observation that most of the salient objects may stand out at least in one modality, this paper proposes an adaptive fusion scheme to fuse saliency predictions generated from two modalities. Specifically, we design a two-streamed convolutional neural network (CNN), each of which extracts features and predicts a saliency map from either RGB or depth modality. Then, a saliency fusion module learns a switch map that is used to adaptively fuse the predicted salie…
UNet-based methods have shown outstanding performance in salient object detection (SOD), but are problematic in two aspects. 1) Indiscriminately integrating the encoder feature, which contains spatial information for multiple objects, and the decoder feature, which contains global information of the salient object, is likely to convey unnecessary details of non-salient objects to the decoder, hindering saliency detection. 2) To deal with ambiguous object boundaries and generate accurate saliency maps, the model needs additional branches, such as edge reconstructions, which leads to increasing …
Bosch Center for Artificial Intelligence(博世人工智能中心)
Recently, there has been a growing interest in developing saliency methods that provide visual explanations of network predictions. Still, the usability of existing methods is limited to image classification models. To overcome this limitation, we extend the existing approaches to generate grid saliencies, which provide spatially coherent visual explanations for (pixel-level) dense prediction networks. As the proposed grid saliency allows to spatially disentangle the object and its context, we specifically explore its potential to produce context explanations for semantic segmentation networks…
Tianjin University(天津大学);Chinese Academy of Sciences(中国科学院)
Stereoscopic perception is an important part of human visual system that allows the brain to perceive depth. However, depth information has not been well explored in existing saliency detection models. In this letter, a novel saliency detection method for stereoscopic images is proposed. Firstly, we propose a measure to evaluate the reliability of depth map, and use it to reduce the influence of poor depth map on saliency detection. Then, the input image is represented as a graph, and the depth information is introduced into graph construction. After that, a new definition of compactness using…
Weakly supervised semantic segmentation (WSSS) using only image-level labels can greatly reduce the annotation cost and therefore has attracted considerable research interest. However, its performance is still inferior to the fully supervised counterparts. To mitigate the performance gap, we propose a saliency guided self-attention network (SGAN) to address the WSSS problem. The introduced self-attention mechanism is able to capture rich and extensive contextual information but may mis-spread attentions to unexpected regions. In order to enable this mechanism to work effectively under weak sup…
Various saliency detection algorithms from color images have been proposed to mimic eye fixation or attentive object detection response of human observers for the same scenes. However, developments on hyperspectral imaging systems enable us to obtain redundant spectral information of the observed scenes from the reflected light source from objects. A few studies using low-level features on hyperspectral images demonstrated that salient object detection can be achieved. In this work, we proposed a salient object detection model on hyperspectral images by applying manifold ranking (MR) on self-s…
Existing studies in weakly-supervised semantic segmentation (WSSS) using image-level weak supervision have several limitations: sparse object coverage, inaccurate object boundaries, and co-occurring pixels from non-target objects. To overcome these challenges, we propose a novel framework, namely Explicit Pseudo-pixel Supervision (EPS), which learns from pixel-level feedback by combining two weak supervisions; the image-level label provides the object identity via the localization map and the saliency map from the off-the-shelf saliency detection model offers rich boundaries. We devise a joint…
University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Virginia Tech(弗吉尼亚理工学院暨州立大学)
Unsupervised video segmentation plays an important role in a wide variety of applications from object identification to compression. However, to date, fast motion, motion blur and occlusions pose significant challenges. To address these challenges for unsupervised video segmentation, we develop a novel saliency estimation technique as well as a novel neighborhood graph, based on optical flow and edge cues. Our approach leads to significantly better initial foreground-background estimates and their robust as well as accurate diffusion across time. We evaluate our proposed algorithm on the chall…
Medical image segmentation models are typically supervised by expert annotations at the pixel-level, which can be expensive to acquire. In this work, we propose a method that combines the high quality of pixel-level expert annotations with the scale of coarse DNN-generated saliency maps for training multi-label semantic segmentation models. We demonstrate the application of our semi-supervised method, which we call CheXseg, on multi-label chest X-ray interpretation. We find that CheXseg improves upon the performance (mIoU) of fully-supervised methods that use only pixel-level expert annotation…
DiSa: Saliency-Aware Foreground-Background Disentangled Framework for Open-Vocabulary Semantic Segmentation
Department of Computer Science and Engineering, Lehigh University(计算机科学与工程系,莱恩大学);Department of Computer Science(计算机科学系);Engineering, Lehigh University(工程系,莱恩大学);Qualcomm AI Research(高通人工智能研究)
In this paper, we present a novel model for simultaneous stable co-saliency detection (CoSOD) and object co-segmentation (CoSEG). To detect co-saliency (segmentation) accurately, the core problem is to well model inter-image relations between an image group. Some methods design sophisticated modules, such as recurrent neural network (RNN), to address this problem. However, order-sensitive problem is the major drawback of RNN, which heavily affects the stability of proposed CoSOD (CoSEG) model. In this paper, inspired by RNN-based model, we first propose a multi-path stable recurrent unit (MSRU…
Relevance-guided Audio Visual Fusion for Video Saliency Prediction
School of Computer Science, Nanjing University of Information Science and Technology(南京信息工程大学计算机科学学院);Jiangsu Collaborative Innovation Center of Atmospheric Environment and Equipment Technology, Nanjing University of Information Science and Technology(南京信息工程大学江苏省大气环境与装备技术协同创新中心);College of Computer and Technology, Nanjing University of Aeronautics and Astronautics(南京航空航天大学计算机科学与技术学院)
City University of Hong Kong(香港城市大学);Shenyang Institute of Automation, Chinese Academy of Sciences(中国科学院沈阳自动化研究所);University of Chinese Academy of Sciences(中国科学院大学)
Numerous efforts have been made to design different low level saliency cues for the RGBD saliency detection, such as color or depth contrast features, background and color compactness priors. However, how these saliency cues interact with each other and how to incorporate these low level saliency cues effectively to generate a master saliency map remain a challenging problem. In this paper, we design a new convolutional neural network (CNN) to fuse different low level saliency cues into hierarchical features for automatically detecting salient objects in RGBD images. In contrast to the existin…
Weakly supervised semantic segmentation with only image-level labels aims to reduce annotation costs for the segmentation task. Existing approaches generally leverage class activation maps (CAMs) to locate the object regions for pseudo label generation. However, CAMs can only discover the most discriminative parts of objects, thus leading to inferior pixel-level pseudo labels. To address this issue, we propose a saliency guided Inter- and Intra-Class Relation Constrained (I$^2$CRC) framework to assist the expansion of the activated object regions in CAMs. Specifically, we propose a saliency gu…
Dalian University of Technology(大连理工大学);UC Berkeley(加州大学伯克利分校);York University(约克大学)
RGB-D saliency detection integrates information from both RGB images and depth maps to improve prediction of salient regions under challenging conditions. The key to RGB-D saliency detection is to fully mine and fuse information at multiple scales across the two modalities. Previous approaches tend to apply the multi-scale and multi-modal fusion separately via local operations, which fails to capture long-range dependencies. Here we propose a transformer-based network to address this issue. Our proposed architecture is composed of two modules: a transformer-based within-modality feature enhanc…
Coronavirus Disease 2019 (COVID-19) has rapidly spread in 2020, emerging a mass of studies for lung infection segmentation from CT images. Though many methods have been proposed for this issue, it is a challenging task because of infections of various size appearing in different lobe zones. To tackle these issues, we propose a Graph-based Pyramid Global Context Reasoning (Graph-PGCR) module, which is capable of modeling long-range dependencies among disjoint infections as well as adapt size variation. We first incorporate graph convolution to exploit long-term contextual information from multi…
University of Exeter(埃克塞特大学);Department of Computer Science University of Exeter(埃克塞特大学计算机科学系)
Saliency prediction is a well studied problem in computer vision. Early saliency models were based on low-level hand-crafted feature derived from insights gained in neuroscience and psychophysics. In the wake of deep learning breakthrough, a new cohort of models were proposed based on neural network architectures, allowing significantly higher gaze prediction than previous shallow models, on all metrics. However, most models treat the saliency prediction as a \textit{regression} problem, and accurate regression of high-dimensional data is known to be a hard problem. Furthermore, it is unclear …
The University of Sydney(悉尼大学);Murdoch University(莫道克大学);Hong Kong University of Sciences and Technology(香港科技大学);The University of Western Australia(西澳大学)
Semantic segmentation is a challenging task in the absence of densely labelled data. Only relying on class activation maps (CAM) with image-level labels provides deficient segmentation supervision. Prior works thus consider pre-trained models to produce coarse saliency maps to guide the generation of pseudo segmentation labels. However, the commonly used off-line heuristic generation process cannot fully exploit the benefits of these coarse saliency maps. Motivated by the significant inter-task correlation, we propose a novel weakly supervised multi-task framework termed as AuxSegNet, to lever…
AI Technology Lab, OPEN8 Singapore(AI技术实验室,OPEN8新加坡)
Contour information plays a vital role in salient object detection. However, excessive false positives remain in predictions from existing contour-based models due to insufficient contour-saliency fusion. In this work, we designed a network for better edge quality in salient object detection. We proposed a contour-saliency blending module to exchange information between contour and saliency. We adopted recursive CNN to increase contour-saliency fusion while keeping the total trainable parameters the same. Furthermore, we designed a stage-wise feature extraction module to help the model pick up…
SalFAU-Net: Saliency Fusion Attention U-Net for Salient Object Detection
School of Electronic and Information Engineering, China West Normal University(西华师范大学电子与信息工程学院);Paul C. Lauterbur Research Center, Shenzhen Institute of Advanced Technology, Chinese Academy of Science(中国科学院深圳先进技术研究院保罗·C·劳特伯研究中心);School of Computer Science, China West Normal University(西华师范大学计算机学院)
RGB-D salient object detection (SOD) is usually formulated as a problem of classification or regression over two modalities, i.e., RGB and depth. Hence, effective RGBD feature modeling and multi-modal feature fusion both play a vital role in RGB-D SOD. In this paper, we propose a depth-sensitive RGB feature modeling scheme using the depth-wise geometric prior of salient objects. In principle, the feature modeling scheme is carried out in a depth-sensitive attention module, which leads to the RGB feature enhancement as well as the background distraction reduction by capturing the depth geometry…
Sun Yat-sen University(中山大学);The University of Hong Kong(香港大学)
Image saliency detection has recently witnessed rapid progress due to deep convolutional neural networks. However, none of the existing methods is able to identify object instances in the detected salient regions. In this paper, we present a salient instance segmentation method that produces a saliency mask with distinct object instance labels for an input image. Our method consists of three steps, estimating saliency map, detecting salient object contours and identifying salient object instances. For the first two steps, we propose a multiscale saliency refinement network, which generates hig…
Peking University(北京大学);The Johns Hopkins University(约翰霍普金斯大学);The Johns Hopkins Medical Institute(约翰霍普金斯医学院)
We aim at segmenting small organs (e.g., the pancreas) from abdominal CT scans. As the target often occupies a relatively small region in the input image, deep neural networks can be easily confused by the complex and variable background. To alleviate this, researchers proposed a coarse-to-fine approach, which used prediction from the first (coarse) stage to indicate a smaller input region for the second (fine) stage. Despite its effectiveness, this algorithm dealt with two stages individually, which lacked optimizing a global energy function, and limited its ability to incorporate multi-stage…
School of Computer Science, Wuhan University(武汉大学计算机学院);School of Computer Science, The University of Sydney(悉尼大学计算机学院);School of Computer Science and Engineering, Nanyang Technological University(南洋理工大学计算机科学与工程学院)
South China University of Technology(华南理工大学);Nanyang Technological University(南洋理工大学)
Humans tend to mine objects by learning from a group of images or several frames of video since we live in a dynamic world. In the computer vision area, many researches focus on co-segmentation (CoS), co-saliency detection (CoSD) and video salient object detection (VSOD) to discover the co-occurrent objects. However, previous approaches design different networks on these similar tasks separately, and they are difficult to apply to each other, which lowers the upper bound of the transferability of deep learning frameworks. Besides, they fail to take full advantage of the cues among inter- and i…
Northwestern Polytechnical University(西北工业大学);Zhejiang Lab(之江实验室);Inception Institute of Artificial Intelligence(起源人工智能研究院);University of Chinese Academy of Sciences(中国科学院大学);NetEase(网易);Huazhong University of Science and Technology(华中科技大学)
Recently, inspired by DETR variants, query-based end-to-end instance segmentation (QEIS) methods have outperformed CNN-based models on large-scale datasets. Yet they would lose efficacy when only a small amount of training data is available since it's hard for the crucial queries/kernels to learn localization and shape priors. To this end, this work offers a novel unsupervised pre-training solution for low-data regimes. Inspired by the recent success of the Prompting technique, we introduce a new pre-training method that boosts QEIS models by giving Saliency Prompt for queries/kernels. Our met…
Max Planck Institute for Informatics(马克斯·普朗克计算机科学研究所);Amsterdam Machine Learning Lab(阿姆斯特丹机器学习实验室);Saarland Informatics Campus(萨尔兰信息学园区);University of Amsterdam(阿姆斯特丹大学)
There have been remarkable improvements in the semantic labelling task in the recent years. However, the state of the art methods rely on large-scale pixel-level annotations. This paper studies the problem of training a pixel-wise semantic labeller network from image-level annotations of the present object classes. Recently, it has been shown that high quality seeds indicating discriminative object regions can be obtained from image-level labels. Without additional information, obtaining the full extent of the object is an inherently ill-posed problem due to co-occurrences. We propose using a …