Towards Accurate Camouflaged Object Detection with Mixture Convolution and Interactive Fusion
National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, School of Computer Science and Engineering, Northwestern Polytechnical University（西北工业大学计算机学院）；Center for Brain Imaging Science and Technology, Zhejiang University（浙江大学）；School of Computer Science, China University of Geosciences（中国地质大学计算机学院）；Optical Electrical and Computer Engineering, University of Shanghai for Science and Technology（上海理工大学）；School of Electrical Engineering & Information Department, Northeast Petroleum University（东北石油大学）；School of Electronic, Electrical Engineering and Physics, Fujian University of Technology（福建工程学院）
Camouflaged object detection (COD), which aims to identify the objects that conceal themselves into the surroundings, has recently drawn increasing research efforts in the field of computer vision. In practice, the success of deep learning based COD is mainly determined by two key factors, including (i) A significantly large receptive field, which provides rich context information, and (ii) An effective fusion strategy, which aggregates the rich multi-level features for accurate COD. Motivated by these observations, in this paper, we propose a novel deep learning based COD approach, which integrates the large receptive field and effective feature fusion into a unified framework. Specifically, we first extract multi-level features from a backbone network. The resulting features are then fed to the proposed dual-branch mixture convolution modules, each of which utilizes multiple asymmetric convolutional layers and two dilated convolutional layers to extract rich context features from a large receptive field. Finally, we fuse the features using specially-designed multilevel interactive fusion modules, each of which employs an attention mechanism along with feature interaction for effective feature fusion. Our method detects camouflaged objects with an effective fusion strategy, which aggregates the rich context information from a large receptive field. All of these designs meet the requirements of COD well, allowing the accurate detection of camouflaged objects. Extensive experiments on widely-used benchmark datasets demonstrate that our method is capable of accurately detecting camouflaged objects and outperforms the state-of-the-art methods.
Camouflaged object detection intends to discover the concealed objects hidden in the surroundings. Existing methods follow the bio-inspired framework, which first locates the object and second refines the boundary. We argue that the discovery of camouflaged objects depends on the recurrent search for the object and the boundary. The recurrent processing makes the human tired and helpless, but it is just the advantage of the transformer with global search ability. Therefore, a dual-task interactive transformer is proposed to detect both accurate position of the camouflaged object and its detail…
Explicit Motion Handling and Interactive Prompting for Video Camouflaged Object Detection
National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University（合成视觉基础科学国家重点实验室，四川大学）；College of Computer Science（计算机学院）；School of Computing（计算学院）；Australian National University（澳大利亚国立大学）
South China University of Technology（华南理工大学）；The Chinese University of Hong Kong（香港中文大学）；The Open University of Hong Kong（香港公开大学）
Camouflaged object detection is a challenging task that aims to identify objects having similar texture to the surroundings. This paper presents to amplify the subtle texture difference between camouflaged objects and the background for camouflaged object detection by formulating multiple texture-aware refinement modules to learn the texture-aware features in a deep convolutional neural network. The texture-aware refinement module computes the covariance matrices of feature responses to extract the texture information, designs an affinity loss to learn a set of parameter maps that help to sepa…
We present OSFormer, the first one-stage transformer framework for camouflaged instance segmentation (CIS). OSFormer is based on two key designs. First, we design a location-sensing transformer (LST) to obtain the location label and instance-aware parameters by introducing the location-guided queries and the blend-convolution feedforward network. Second, we develop a coarse-to-fine fusion (CFF) to merge diverse context information from the LST encoder and CNN backbone. Coupling these two components enables OSFormer to efficiently blend local features and long-range context dependencies for pre…
University of Chinese Academy of Sciences（中国科学院大学）；Sun Yat-Sen University（中山大学）；Tsinghua University（清华大学）
Camouflaged objects are seamlessly blended in with their surroundings, which brings a challenging detection task in computer vision. Optimizing a convolutional neural network (CNN) for camouflaged object detection (COD) tends to activate local discriminative regions while ignoring complete object extent, causing the partial activation issue which inevitably leads to missing or redundant regions of objects. In this paper, we argue that partial activation is caused by the intrinsic characteristics of CNN, where the convolution operations produce local receptive fields and experience difficulty t…
Referring Camouflaged Object Detection With Multi-Context Overlapped Windows Cross-Attention
School of Computer and Information Engineering, Shanghai Polytechnic University（计算机与信息工程学院，上海理工大学）；Shanghai Key Laboratory of Intelligent Information Processing, School of Computer Science, Fudan University（智能信息处理重点实验室，复旦大学）；Department of Automation, School of Intelligent Manufacturing and Control Engineering, Shanghai Polytechnic University（自动化系，智能制造与控制工程学院，上海理工大学）；Academy for Engineering and Technology, Fudan University（工程与技术学院，复旦大学）；Research and Design Center, Shanghai Insititue Of Computer Technology Company（研发设计中心，上海计算机技术有限公司）
High-Resolution Underwater Camouflaged Object Detection: GBU-UCOD Dataset and Topology-Aware and Frequency-Decoupled Networks
College of Shipbuilding Engineering, Harbin Engineering University（哈尔滨工程大学船舶工程学院）；School of Information Science and Technology, Great Bay University（广东工业大学信息科学与技术学院）
In this paper, we propose a novel fully convolutional two-stream fusion network (FCTSFN) for interactive image segmentation. The proposed network includes two sub-networks: a two-stream late fusion network (TSLFN) that predicts the foreground at a reduced resolution, and a multi-scale refining network (MSRN) that refines the foreground at full resolution. The TSLFN includes two distinct deep streams followed by a fusion network. The intuition is that, since user interactions are more direct information on foreground/background than the image itself, the two-stream structure of the TSLFN reduce…
Let Human Sketches Help: Empowering Challenging Image Segmentation Task with Freehand Sketches
School of Information Engineering, Huzhou University（湖州大学信息工程学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Software Technology, Zhejiang University（浙江大学软件技术学院）；Information Systems Technology and Design Pillar, Singapore University of Technology and Design（新加坡科技设计大学信息系统技术与设计 pillar（注：此处保留原文结构，因无通用标准译法））；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）；KOKONI, Moxin (Huzhou) Tech. Co., LTD.（摩芯（湖州）科技有限公司 KOKONI）；Institute of Psychology, Chinese Academy of Sciences（中国科学院心理研究所）
Deep learning techniques have successfully been employed in numerous computer vision tasks including image segmentation. The techniques have also been applied to medical image segmentation, one of the most critical tasks in computer-aided diagnosis. Compared with natural images, the medical image is a gray-scale image with low-contrast (even with some invisible parts). Because some organs have similar intensity and texture with neighboring organs, there is usually a need to refine automatic segmentation results. In this paper, we propose an interactive deep refinement framework to improve the …
University of Dayton（代顿大学）；Institute of Industrial Science, the University of Tokyo（东京大学生产技术研究所）；The Graduate University for Advanced Studies (SOKENDAI)（综合研究大学院大学）；University of Science, VNU-HCM（越南国立大学胡志明市分校科学大学）；National Institute of Informatics（国立信息学研究所）
Camouflaged objects attempt to conceal their texture into the background and discriminating them from the background is hard even for human beings. The main objective of this paper is to explore the camouflaged object segmentation problem, namely, segmenting the camouflaged object(s) for a given image. This problem has not been well studied in spite of a wide range of potential applications including the preservation of wild animals and the discovery of new species, surveillance systems, search-and-rescue missions in the event of natural disasters such as earthquakes, floods or hurricanes. Thi…
With recent innovations in dense image captioning, it is now possible to describe every object of the scene with a caption while objects are determined by bounding boxes. However, interpretation of such an output is not trivial due to the existence of many overlapping bounding boxes. Furthermore, in current captioning frameworks, the user is not able to involve personal preferences to exclude out of interest areas. In this paper, we propose a novel hybrid deep learning architecture for interactive region segmentation and captioning where the user is able to specify an arbitrary region of the i…
Interactive Image Selection and Training for Brain Tumor Segmentation Network
Institute of Computing, University of Campinas（坎皮纳斯大学计算研究所）；Hospital de Clínicas, Universidade Federal do Paraná（巴拉那联邦大学临床医院）；Instituto de Neurologia de Curitiba（库里蒂巴神经研究所）
Ocean University of China（中国海洋大学）；Technical University of Munich（慕尼黑工业大学）；University of Warwick（华威大学）；University of Leicester（莱斯特大学）；University of Southampton（南安普顿大学）
Interactive single-image segmentation is ubiquitous in the scientific and commercial imaging software. In this work, we focus on the single-image segmentation problem only with some seeds such as scribbles. Inspired by the dynamic receptive field in the human being's visual system, we propose the Gaussian dynamic convolution (GDC) to fast and efficiently aggregate the contextual information for neural networks. The core idea is randomly selecting the spatial sampling area according to the Gaussian distribution offsets. Our GDC can be easily used as a module to build lightweight or complex segm…
Salient Object Detection is the task of predicting the human attended region in a given scene. Fusing depth information has been proven effective in this task. The main challenge of this problem is how to aggregate the complementary information from RGB modality and depth modality. However, conventional deep models heavily rely on CNN feature extractors, and the long-range contextual dependencies are usually ignored. In this work, we propose Dual Swin-Transformer based Mutual Interactive Network. We adopt Swin-Transformer as the feature extractor for both RGB and depth modality to model the lo…
University College London（伦敦大学学院）；KU Leuven（鲁汶大学）；University Hospitals KU Leuven（鲁汶大学医院）
Convolutional neural networks (CNNs) have achieved state-of-the-art performance for automatic medical image segmentation. However, they have not demonstrated sufficiently accurate and robust results for clinical use. In addition, they are limited by the lack of image-specific adaptation and the lack of generalizability to previously unseen object classes. To address these problems, we propose a novel deep learning-based framework for interactive segmentation by incorporating CNNs into a bounding box and scribble-based segmentation pipeline. We propose image-specific fine-tuning to make a CNN m…
Maybe you don't need a U-Net: convolutional feature upsampling for materials micrograph segmentation
Department of Materials, Imperial College London（材料系，帝国理工学院伦敦分校）；Dyson School of Design Engineering, Imperial College London（设计工程学院，帝国理工学院伦敦分校）；Finden ltd（Finden公司）；Rutherford Appleton Laboratory, Harwell Science and Innovation Campus（拉瑟福德-苹果顿实验室，哈罗尔德科学与创新校园）
School of Computer Science and Statistics（计算机科学与统计学学院）；Lero, Trinity College Dublin（都柏林圣三一学院Lero（爱尔兰国家软件与计算机科学研究所））
Convolutional neural networks (CNNs) have dramatically improved the accuracy of tasks such as object recognition, image segmentation and interactive speech systems. CNNs require large amounts of computing resources because ofcomputationally intensive convolution layers. Fast convolution algorithms such as Winograd convolution can greatly reduce the computational cost of these layers at a cost of poor numeric properties, such that greater savings in computation exponentially increase floating point errors. A defining feature of each Winograd convolution algorithm is a set of real-value points w…
VFM-ISRefiner: Towards Better Adapting Vision Foundation Models for Interactive Segmentation of Remote Sensing Images
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院航天信息研究所）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）；School of Information Science and Technology, Beijing Forestry University（北京林业大学信息科学与技术学院）
University of Chinese Academy of Sciences（中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）；Huawei Inc.（华为公司）
Within Convolutional Neural Network (CNN), the convolution operations are good at extracting local features but experience difficulty to capture global representations. Within visual transformer, the cascaded self-attention modules can capture long-distance feature dependencies but unfortunately deteriorate local feature details. In this paper, we propose a hybrid network structure, termed Conformer, to take advantage of convolutional operations and self-attention mechanisms for enhanced representation learning. Conformer roots in the Feature Coupling Unit (FCU), which fuses local features and…
I-MPN: Inductive Message Passing Network for Efficient Human-in-the-Loop Annotation of Mobile Eye Tracking Data
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；University of Science, VNU-HCM（胡志明市国家大学科学大学）；Quy Nhon AI Research and Development Center, FPT Software（FPT软件归仁人工智能研发中心）；Max Planck Research School for Intelligent Systems (IMPRS-IS)（马克斯·普朗克智能系统研究院）
BihoT: A Large-Scale Dataset and Benchmark for Hyperspectral Camouflaged Object Tracking
School of Information and Electronics, Beijing Institute of Technology（信息与电子学院，北京理工大学）；Beijing Key Laboratory of Fractional Signals and Systems（北京分数信号与系统重点实验室）；National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing（空间生智能信息处理国家重点实验室）；Department of Electrical and Computer Engineering, University of Delaware（电气与计算机工程系，德雷塞尔大学）；Department of Electrical and Computer Engineering, Mississippi State University（电气与计算机工程系，密苏里州立大学）
University Hospitals Leuven（鲁汶大学医院）；King’s College London（伦敦国王学院）；KU Leuven（鲁汶大学）；INESC TEC - Institute for Systems and Computer Engineering, Technology and Science（INESC TEC - 系统与计算机工程、技术与科学研究所）
Automatic medical image segmentation via convolutional neural networks (CNNs) has shown promising results. However, they may not always be robust enough for clinical use. Sub-optimal segmentation would require clinician's to manually delineate the target object, causing frustration. To address this problem, a novel interactive CNN-based segmentation framework is proposed in this work. The aim is to represent the CNN segmentation contour as B-splines by utilising B-spline explicit active surfaces (BEAS). The interactive element of the framework allows the user to precisely edit the contour in r…
Friedrich-Alexander University Erlangen-Nuremberg（埃尔朗根-纽伦堡大学）；Graduate School in Advanced Optical Technologies (SAOT)（先进光学技术研究生院）；Siemens Healthcare GmbH（西门子医疗有限公司）
For complex segmentation tasks, fully automatic systems are inherently limited in their achievable accuracy for extracting relevant objects. Especially in cases where only few data sets need to be processed for a highly accurate result, semi-automatic segmentation techniques exhibit a clear benefit for the user. One area of application is medical image processing during an intervention for a single patient. We propose a learning-based cooperative segmentation approach which includes the computing entity as well as the user into the task. Our system builds upon a state-of-the-art fully convolut…
Université de Lyon（里昂大学）；CNRS（法国国家科学研究中心）；École Centrale de Lyon（里昂中央理工学院）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Inria（法国国家信息与自动化研究所）；Grenoble INP（格勒诺布尔国立高等物理、数学与信息学院）；LIRIS（图像与信息系统科学实验室）；LIG（格勒诺布尔信息学实验室）；Siléane（Siléane公司）
With more and more household objects built on planned obsolescence and consumed by a fast-growing population, hazardous waste recycling has become a critical challenge. Given the large variability of household waste, current recycling platforms mostly rely on human operators to analyze the scene, typically composed of many object instances piled up in bulk. Helping them by robotizing the unitary extraction is a key challenge to speed up this tedious process. Whereas supervised deep learning has proven very efficient for such object-level scene understanding, e.g., generic object detection and …
Harbin Engineering University（哈尔滨工程大学）；CAST（中国航天科技集团公司）；University of Surrey（萨里大学）；University of Leicester（莱斯特大学）
Object detection in aerial images is a challenging task due to the lack of visible features and variant orientation of objects. Significant progress has been made recently for predicting targets from aerial images with horizontal bounding boxes (HBBs) and oriented bounding boxes (OBBs) using two-stage detectors with region based convolutional neural networks (R-CNN), involving object localization in one stage and object classification in the other. However, the computational complexity in two-stage detectors is often high, especially for orientational object detection, due to anchor matching a…
Human-Object Interaction (HOI) detection devotes to learn how humans interact with surrounding objects via inferring triplets of < human, verb, object >. However, recent HOI detection methods mostly rely on additional annotations (e.g., human pose) and neglect powerful interactive reasoning beyond convolutions. In this paper, we present a novel graph-based interactive reasoning model called Interactive Graph (abbr. in-Graph) to infer HOIs, in which interactive semantics implied among visual targets are efficiently exploited. The proposed model consists of a project function that maps related t…
Deep learning requires large amounts of training data to be effective. For the task of object segmentation, manually labeling data is very expensive, and hence interactive methods are needed. Following recent approaches, we develop an interactive object segmentation system which uses user input in the form of clicks as the input to a convolutional network. While previous methods use heuristic click sampling strategies to emulate user clicks during training, we propose a new iterative training strategy. During training, we iteratively add clicks based on the errors of the currently predicted se…
We provide a high fidelity deep learning algorithm (HyperSeg) for interactive video segmentation tasks using a convolutional network with context-aware skip connections, and compressed, hypercolumn image features combined with a convolutional tessellation procedure. In order to maintain high output fidelity, our model crucially processes and renders all image features in high resolution, without utilizing downsampling or pooling procedures. We maintain this consistent, high grade fidelity efficiently in our model chiefly through two means: (1) We use a statistically-principled tensor decomposi…
Pixelwise annotation of image sequences can be very tedious for humans. Interactive video object segmentation aims to utilize automatic methods to speed up the process and reduce the workload of the annotators. Most contemporary approaches rely on deep convolutional networks to collect and process information from human annotations throughout the video. However, such networks contain millions of parameters and need huge amounts of labeled training data to avoid overfitting. Beyond that, label propagation is usually executed as a series of frame-by-frame inference steps, which is difficult to b…
Microsoft Research（微软研究院）；Johns Hopkins University（约翰斯·霍普金斯大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）
An increasing number of computer vision tasks can be tackled with deep features, which are the intermediate outputs of a pre-trained Convolutional Neural Network. Despite the astonishing performance, deep features extracted from low-level neurons are still below satisfaction, arguably because they cannot access the spatial context contained in the higher layers. In this paper, we present InterActive, a novel algorithm which computes the activeness of neurons and network connections. Activeness is propagated through a neural network in a top-down manner, carrying high-level context and improvin…
Multi-view Graph Convolutional Network with Fully Leveraging Consistency via Granular-ball-based Topology Construction, Feature Enhancement and Interactive Fusion
School of Computer, Jiangsu University of Science（江苏科技大学计算机学院）；Chongqing Key Laboratory of Computational Intelligence, Chongqing University of Posts（重庆邮电大学计算智能重点实验室）；College of Computer（计算机学院）；Data Science, Fuzhou University, Fuzhou 350116, China（福州大学数据科学学院）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Adobe Research（Adobe研究院）
Interactive object selection is a very important research problem and has many applications. Previous algorithms require substantial user interactions to estimate the foreground and background distributions. In this paper, we present a novel deep learning based algorithm which has a much better understanding of objectness and thus can reduce user interactions to just a few clicks. Our algorithm transforms user provided positive and negative clicks into two Euclidean distance maps which are then concatenated with the RGB channels of images to compose (image, user interactions) pairs. We generat…
We present a deep learning method for the interactive video object segmentation. Our method is built upon two core operations, interaction and propagation, and each operation is conducted by Convolutional Neural Networks. The two networks are connected both internally and externally so that the networks are trained jointly and interact with each other to solve the complex video object segmentation problem. We propose a new multi-round training scheme for the interactive video object segmentation so that the networks can learn how to understand the user's intention and update incorrect estimati…
In interactive object segmentation a user collaborates with a computer vision model to segment an object. Recent works employ convolutional neural networks for this task: Given an image and a set of corrections made by the user as input, they output a segmentation mask. These approaches achieve strong performance by training on large datasets but they keep the model parameters unchanged at test time. Instead, we recognize that user corrections can serve as sparse training examples and we propose a method that capitalizes on that idea to update the model parameters on-the-fly to the data at han…
University College London（伦敦大学学院）；University Hospitals KU Leuven（鲁汶大学附属医院）
Accurate medical image segmentation is essential for diagnosis, surgical planning and many other applications. Convolutional Neural Networks (CNNs) have become the state-of-the-art automatic segmentation methods. However, fully automatic results may still need to be refined to become accurate and robust enough for clinical use. We propose a deep learning-based interactive segmentation method to improve the results obtained by an automatic CNN and to reduce user interactions during refinement for higher accuracy. We use one CNN to obtain an initial automatic segmentation, on which user interact…
University of Electronic Science and Technology of China（电子科技大学）；SenseTime Research（商汤科技研究院）；Shanghai Jiao Tong University（上海交通大学）；University Hospitals Leuven（鲁汶大学医院）；University College London（伦敦大学学院）；King’s College London（伦敦国王学院）
Segmentation of organs or lesions from medical images plays an essential role in many clinical applications such as diagnosis and treatment planning. Though Convolutional Neural Networks (CNN) have achieved the state-of-the-art performance for automatic segmentation, they are often limited by the lack of clinically acceptable accuracy and robustness in complex cases. Therefore, interactive segmentation is a practical alternative to these methods. However, traditional interactive segmentation methods require a large amount of user interactions, and recently proposed CNN-based interactive segmen…
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；NVIDIA（英伟达）
Manually labeling objects by tracing their boundaries is a laborious process. In Polygon-RNN++ the authors proposed Polygon-RNN that produces polygonal annotations in a recurrent manner using a CNN-RNN architecture, allowing interactive correction via humans-in-the-loop. We propose a new framework that alleviates the sequential nature of Polygon-RNN, by predicting all vertices simultaneously using a Graph Convolutional Network (GCN). Our model is trained end-to-end. It supports object annotation by either polygons or splines, facilitating labeling efficiency for both line-based and curved obje…
Johns Hopkins University（约翰霍普金斯大学）；Technische Universität München（慕尼黑工业大学）
We propose a method for interactive boundary extraction which combines a deep, patch-based representation with an active contour framework. We train a class-specific convolutional neural network which predicts a vector pointing from the respective point on the evolving contour towards the closest point on the boundary of the object of interest. These predictions form a vector field which is then used for evolving the contour by the Sobolev active contour framework proposed by Sundaramoorthi et al. The resulting interactive segmentation method is very efficient in terms of required computationa…
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Nvidia（英伟达）
Inferring detailed 3D geometry of the scene is crucial for robotics applications, simulation, and 3D content creation. However, such information is hard to obtain, and thus very few datasets support it. In this paper, we propose an interactive framework for annotating 3D object geometry from both point cloud data and RGB imagery. The key idea behind our approach is to exploit strong priors that humans have about the 3D world in order to interactively annotate complete 3D shapes. Our framework targets naive users without artistic or graphics expertise. We introduce two simple-to-use interaction…
Fine-grained Visual Classification (FGVC) aims to identify objects from subcategories. It is a very challenging task because of the subtle inter-class differences. Existing research applies large-scale convolutional neural networks or visual transformers as the feature extractor, which is extremely computationally expensive. In fact, real-world scenarios of fine-grained recognition often require a more lightweight mobile network that can be utilized offline. However, the fundamental mobile network feature extraction capability is weaker than large-scale models. In this paper, based on the ligh…
