Supervised object detection and semantic segmentation require object or even pixel level annotations. When there exist image level labels only, it is challenging for weakly supervised algorithms to achieve accurate predictions. The accuracy achieved by top weakly supervised algorithms is still significantly lower than their fully supervised counterparts. In this paper, we propose a novel weakly supervised curriculum learning pipeline for multi-label object recognition, detection and semantic segmentation. In this pipeline, we first obtain intermediate object localization and pixel labeling results for the training images, and then use such results to train task-specific deep networks in a fully supervised manner. The entire process consists of four stages, including object localization in the training images, filtering and fusing object instances, pixel labeling for the training images, and task-specific network training. To obtain clean object instances in the training images, we propose a novel algorithm for filtering, fusing and classifying object instances collected from multiple solution mechanisms. In this algorithm, we incorporate both metric learning and density-based clustering to filter detected object instances. Experiments show that our weakly supervised pipeline achieves state-of-the-art results in multi-label image classification as well as weakly supervised object detection and very competitive results in weakly supervised semantic segmentation on MS-COCO, PASCAL VOC 2007 and PASCAL VOC 2012.
The current fusion positioning systems are mainly based on filtering algorithms, such as Kalman filtering or particle filtering. However, the system complexity of practical application scenarios is often very high, such as noise modeling in pedestrian inertial navigation systems, or environmental noise modeling in fingerprint matching and localization algorithms. To solve this problem, this paper proposes a fusion positioning system based on deep learning and proposes a transfer learning strategy for improving the performance of neural network models for samples with different distributions. T…
University of British Columbia（不列颠哥伦比亚大学）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）；Alberta Machine Intelligence Institute（阿尔伯塔机器智能研究所）
Semi-weakly supervised semantic segmentation (SWSSS) aims to train a model to identify objects in images based on a small number of images with pixel-level labels, and many more images with only image-level labels. Most existing SWSSS algorithms extract pixel-level pseudo-labels from an image classifier - a very difficult task to do well, hence requiring complicated architectures and extensive hyperparameter tuning on fully-supervised validation sets. We propose a method called prediction filtering, which instead of extracting pseudo-labels, just uses the classifier as a classifier: it ignores…
Cross-modal Offset-guided Dynamic Alignment and Fusion for Weakly Aligned UAV Object Detection
State Key Laboratory of Optical Field Manipulation Science and Technology, Chinese Academy of Sciences（光学场操控科学与技术国家重点实验室，中国科学院）；Institute of Optics and Electronics, Chinese Academy of Sciences（中国科学院光电研究所）
Context Patch Fusion With Class Token Enhancement for Weakly Supervised Semantic Segmentation
School of Cyber Science and Engineering, Wuxi University（网络安全科学与工程学院，无锡大学）；School of Informatics, Xiamen University（信息学院，厦门大学）；School of Computer Science, University of Liverpool（计算机科学学院，利物浦大学）
In this paper we examine the effect of applying ensemble learning to the performance of collaborative filtering methods. We present several systematic approaches for generating an ensemble of collaborative filtering models based on a single collaborative filtering algorithm (single-model or homogeneous ensemble). We present an adaptation of several popular ensemble techniques in machine learning for the collaborative filtering domain, including bagging, boosting, fusion and randomness injection. We evaluate the proposed approach on several types of collaborative filtering base models: k- NN, m…
Region-Aware CAM: High-Resolution Weakly-Supervised Defect Segmentation via Salient Region Perception
School of Instrumentation Science and Engineering, Harbin Institute of Technology（仪器科学与工程学院，哈尔滨工业大学）；Harbin Institute of Technology Suzhou Research Institute（哈尔滨工业大学苏州研究所）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences（中国科学院脑科学与智能技术卓越创新中心）；Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences（中国科学院香港创新研究院）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
To achieve accurate and robust object detection in the real-world scenario, various forms of images are incorporated, such as color, thermal, and depth. However, multimodal data often suffer from the position shift problem, i.e., the image pair is not strictly aligned, making one object has different positions in different modalities. For the deep learning method, this problem makes it difficult to fuse multimodal features and puzzles the convolutional neural network (CNN) training. In this article, we propose a general multimodal detector named aligned region CNN (AR-CNN) to tackle the positi…
Large scale datasets created from user labels or openly available data have become crucial to provide training data for large scale learning algorithms. While these datasets are easier to acquire, the data are frequently noisy and unreliable, which is motivating research on weakly supervised learning techniques. In this paper we propose an iterative learning method that extracts the useful information from a large scale change detection dataset generated from open vector data to train a fully convolutional network which surpasses the performance obtained by naive supervised learning. We also p…
Hefei Institute of Physical Science, Chinese Academy of Sciences（中国科学院合肥物质科学研究院）；University of Science and Technology of China（中国科学技术大学）；Nanyang Technological University（南洋理工大学）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）
Panchromatic (PAN) and multi-spectral (MS) image fusion, named Pan-sharpening, refers to super-resolve the low-resolution (LR) multi-spectral (MS) images in the spatial domain to generate the expected high-resolution (HR) MS images, conditioning on the corresponding high-resolution PAN images. In this paper, we present a simple yet effective \textit{alternating reverse filtering network} for pan-sharpening. Inspired by the classical reverse filtering that reverses images to the status before filtering, we formulate pan-sharpening as an alternately iterative reverse filtering process, which fus…
In remote sensing, image fusion technique is a useful tool used to fuse high spatial resolution panchromatic images (PAN) with lower spatial resolution multispectral images (MS) to create a high spatial resolution multispectral of image fusion (F) while preserving the spectral information in the multispectral image (MS).There are many PAN sharpening techniques or Pixel-Based image fusion techniques that have been developed to try to enhance the spatial resolution and the spectral property preservation of the MS. This paper attempts to undertake the study of image fusion, by using two types of …
Southeast University（东南大学）；Shanghai United-imaging Healthcare Co., Ltd（上海联影医疗科技股份有限公司）；University of Birmingham（伯明翰大学）
In order to improve speckle noise denoising of block matching 3d filtering (BM3D) method, an image frequency-domain multi-layer fusion enhancement method (MLFE-BM3D) based on nonsubsampled contourlet transform (NSCT) has been proposed. The method designs a NSCT hard threshold denoising enhancement to preprocess the image, then uses fusion enhancement in NSCT domain to fuse the preliminary estimation results of images before and after the NSCT hard threshold denoising, finally, BM3D denoising is carried out with the fused image to obtain the final denoising result. Experiments on natural images…
Musaliar College of Engineering and Technology（穆萨利亚尔工程技术学院）
Image Fusion is the process in which core information from a set of component images is merged to form a single image, which is more informative and complete than the component input images in quality and appearance. This paper presents a fast and effective image fusion method for creating high quality fused images by merging component images. In the proposed method, the input image is broken down to a two-scale image representation with a base layer having large scale variations in intensity, and a detail layer containing small scale details. Here fusion of the base and detail layers is imple…
In-loop filtering is used in video coding to process the reconstructed frame in order to remove blocking artifacts. With the development of convolutional neural networks (CNNs), CNNs have been explored for in-loop filtering considering it can be treated as an image de-noising task. However, in addition to being a distorted image, the reconstructed frame is also obtained by a fixed line of block based encoding operations in video coding. It carries coding-unit based coding distortion of some similar characteristics. Therefore, in this paper, we address the filtering problem from two aspects, gl…
WGAST: Weakly-Supervised Generative Network for Daily 10 m Land Surface Temperature Estimation via Spatio-Temporal Fusion
INSA CVL, PRISME UR 4229（INSA CVL，PRISME UR 4229）；Université d’Orléans, PRISME UR 4229（奥尔良大学，PRISME UR 4229）；Université d’Orléans, CEDETE, UR 1210（奥尔良大学，CEDETE，UR 1210）
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
Autonomous UAV racing has recently emerged as an interesting research problem. The dream is to beat humans in this new fast-paced sport. A common approach is to learn an end-to-end policy that directly predicts controls from raw images by imitating an expert. However, such a policy is limited by the expert it imitates and scaling to other environments and vehicle dynamics is difficult. One approach to overcome the drawbacks of an end-to-end policy is to train a network only on the perception task and handle control with a PID or MPC controller. However, a single controller must be extensively …
Triple Spectral Fusion for Sensor-based Human Activity Recognition
School of Electronics and Communication Engineering, the Shenzhen Campus of Sun Yat-sen University（南方科技大学深圳校区电子与通信工程学院）；Aviation University of Air Force（空军航空大学）；College of Computer Science, Chongqing University（重庆大学计算机学院）；Department of Computer Science and Software Engineering, the University of Western Australia（西澳大学计算机科学与软件工程系）
From 2D Images to 3D Model:Weakly Supervised Multi-View Face Reconstruction with Deep Fusion
University of Liverpool（利物浦大学）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）；Ricoh Software Research Center (Beijing) Co., Ltd.（理光软件研究中心（北京）有限公司）；Edinburgh Napier University（爱丁堡龙比亚大学）；Duke Kunshan University（昆山杜克大学）
With the advancement of IoT and artificial intelligence technologies, and the need for rapid application growth in fields such as security entrance control and financial business trade, facial information processing has become an important means for achieving identity authentication and information security. In this paper, we propose a multi-feature fusion algorithm based on integral histograms and a real-time update tracking particle filtering module. First, edge and colour features are extracted, weighting methods are used to weight the colour histogram and edge features to describe facial f…
Hikvision Research Institute, Hangzhou Hikvision Digital Technology Co. Ltd（杭州海康威视数字技术股份有限公司海康研究院）；Shanghai Jiao Tong University（上海交通大学）
3D vehicle detection based on multi-modal fusion is an important task of many applications such as autonomous driving. Although significant progress has been made, we still observe two aspects that need to be further improvement: First, the specific gain that camera images can bring to 3D detection is seldom explored by previous works. Second, many fusion algorithms run slowly, which is essential for applications with high real-time requirements(autonomous driving). To this end, we propose an end-to-end trainable single-stage multi-modal feature adaptive network in this paper, which uses image…
National Institute of Advanced Industrial Science and Technology（产业技术综合研究所）；AIST Tokyo WaterFront（产业技术综合研究所东京临海中心）
Most of the traditional convolutional neural networks (CNNs) implements bottom-up approach (feed-forward) for image classifications. However, many scientific studies demonstrate that visual perception in primates rely on both bottom-up and top-down connections. Therefore, in this work, we propose a CNN network with feedback structure for Solar power plant detection on middle-resolution satellite images. To express the strength of the top-down connections, we introduce feedback CNN network (FB-Net) to a baseline CNN model used for solar power plant classification on multi-spectral satellite dat…
Beijing University of Posts and Telecommunications（北京邮电大学）；University of Technology Sydney（悉尼科技大学）；Australian AI Institute（澳大利亚人工智能研究所）
For multimodal tasks, a good feature extraction network should extract information as much as possible and ensure that the extracted feature embedding and other modal feature embedding have an excellent mutual understanding. The latter is often more critical in feature fusion than the former. Therefore, selecting the optimal feature extraction network collocation is a very important subproblem in multimodal tasks. Most of the existing studies ignore this problem or adopt an ergodic approach. This problem is modeled as an optimization problem in this paper. A novel method is proposed to convert…
Beijing University of Posts and Telecommunications（北京邮电大学）
Existing Visual Question Answering (VQA) models have explored various visual relationships between objects in the image to answer complex questions, which inevitably introduces irrelevant information brought by inaccurate object detection and text grounding. To address the problem, we propose a Question-Driven Graph Fusion Network (QD-GFN). It first models semantic, spatial, and implicit visual relations in images by three graph attention networks, then question information is utilized to guide the aggregation process of the three graphs, further, our QD-GFN adopts an object filtering mechanis…
Weakly supervised multimodal segmentation of acoustic borehole images with depth-aware cross-attention
Federal University of Bahia, Institute of Geosciences, Department of Geophysics（巴伊亚联邦大学，地质科学学院，地球物理学系）；Grupo de Estudos e Aplicação de Inteligência Artificial em Geofísica (GAIA)（地质物理中人工智能研究与应用小组）
This paper addresses the problem of large-scale image retrieval. We propose a two-layer fusion method which takes advantage of global and local cues and ranks database images from coarse to fine (C2F). Departing from the previous methods fusing multiple image descriptors simultaneously, C2F is featured by a layered procedure composed by filtering and refining. In particular, C2F consists of three components. 1) Distractor filtering. With holistic representations, noise images are filtered out from the database, so the number of candidate images to be used for comparison with the query can be g…
PCA Lab, Nanjing University of Science and Technology, China（南京理工大学科学与工程学院实验室）；National University of Singapore（新加坡国立大学）；University of California, USA, and Yonsei University, South Korea（美国加州大学和韩国延世大学）；PCA Lab, Nanjing University, China（南京大学实验室）
The Chinese University of Hong Kong1,Geng Mei AI lab[2, 4], The University of Hong Kong3.（香港中文大学1，羹美人工智能实验室[2, 4]，香港大学3.）
Fine-grained image classification is a challenging problem, since the difficulty of finding discriminative features. To handle this circumstance, basically, there are two ways to go. One is use attention based method to focus on informative areas, while the other one aims to find high order between features. Further, for attention based method there are two directions, activation based and detection based, which are proved effective by scholars. However ,rare work focus on fusing two types of attention with high order feature. In this paper, we propose a novel DAF method which fuse two types o…
CmFNet: Cross-modal Fusion Network for Weakly-supervised Segmentation of Medical Images
School of Physics, Peking University（北京大学物理学院）；School of Computer Science, Peking University（北京大学计算机学院）；Department of Radiotherapy, Peking University Cancer Hospital（北京大学肿瘤医院放疗科）；Department of Radiation Oncology, Peking University Third Hospital（北京大学第三医院放疗科）
Enhancing Weakly Supervised Multimodal Video Anomaly Detection through Text Guidance
School of Computer Science and Technology, Hangzhou Dianzi University（杭州电子科技大学计算机科学与技术学院）；Alibaba Cloud（阿里云）；College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息科学与电子工程学院）
Model Confidence-Guided Multi-Image Fusion of Fundus Images for Diabetic Retinopathy Diagnosis
Massachusetts Institute of Technology（麻省理工学院）；Wilmer Eye Institute, Department of Ophthalmology, Johns Hopkins University（约翰霍普金斯大学威尔默眼科研究所）；Department of Biomedical Engineering, Johns Hopkins University（约翰霍普金斯大学生物医学工程系）；Bakar Computational Health Sciences Institute, University of California San Francisco（加州大学旧金山分校巴卡计算健康科学研究所）；Department of Ophthalmology, University of California San Francisco（加州大学旧金山分校眼科系）；Division of Clinical Informatics and Digital Transformation, University of California San Francisco（加州大学旧金山分校临床信息学与数字化转型 division）
This paper describes a new method for mitigating the effects of atmospheric distortion on observed sequences that include large moving objects. In order to provide accurate detail from objects behind the distorting layer, we solve the space-variant distortion problem using recursive image fusion based on the Dual Tree Complex Wavelet Transform (DT-CWT). The moving objects are detected and tracked using the improved Gaussian mixture models (GMM) and Kalman filtering. New fusion rules are introduced which work on the magnitudes and angles of the DT-CWT coefficients independently to achieve a sha…
The fusion of camera sensor and inertial data is a leading method for ego-motion tracking in autonomous and smart devices. State estimation techniques that rely on non-linear filtering are a strong paradigm for solving the associated information fusion task. The de facto inference method in this space is the celebrated extended Kalman filter (EKF), which relies on first-order linearizations of both the dynamical and measurement model. This paper takes a critical look at the practical implications and limitations posed by the EKF, especially under faulty visual feature associations and the pres…
Aligning First, Then Fusing: A Novel Weakly Supervised Multimodal Violence Detection Method
School of Software Engineering（软件工程学院）；Shenzhen Institute of Advanced Technology（深圳先进技术研究院）；Xi’an Jiaotong University（西安交通大学）；Chinese Academy of Sciences（中国科学院）
Bilkent University（比尔肯特大学）；Middle East Technical University（中东技术大学）
Multisource image analysis that leverages complementary spectral, spatial, and structural information benefits fine-grained object recognition that aims to classify an object into one of many similar subcategories. However, for multisource tasks that involve relatively small objects, even the smallest registration errors can introduce high uncertainty in the classification process. We approach this problem from a weakly supervised learning perspective in which the input images correspond to larger neighborhoods around the expected object locations where an object with a given class label is pr…
EAR: Enhancing Uni-Modal Representations for Weakly Supervised Audio-Visual Video Parsing
School of Intelligent Engineering and Automation, Beijing University of Posts and Telecommunications（智能工程与自动化学院，北京邮电大学）；State Grid Corporation of China（国家电网公司）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（人工智能学院，北京邮电大学）
