Geo-localization is a critical task in computer vision. In this work, we cast the geo-localization as a 2D image retrieval task. Current state-of-the-art methods for 2D geo-localization are not robust to locate a scene with drastic scale variations because they only exploit features from one semantic level for image representations. To address this limitation, we introduce a hierarchical attention fusion network using multi-scale features for geo-localization. We extract the hierarchical feature maps from a convolutional neural network (CNN) and organically fuse the extracted features for image representations. Our training is self-supervised using adaptive weights to control the attention of feature emphasis from each hierarchical level. Evaluation results on the image retrieval and the large-scale geo-localization benchmarks indicate that our method outperforms the existing state-of-the-art methods. Code is available here: \url{https://github.com/YanLiqi/HAF}.
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
Hierarchical classification is significant for complex tasks by providing multi-granular predictions and encouraging better mistakes. As the label structure decides its performance, many existing approaches attempt to construct an excellent label structure for promoting the classification results. In this paper, we consider that different label structures provide a variety of prior knowledge for category recognition, thus fusing them is helpful to achieve better hierarchical classification results. Furthermore, we propose a multi-task multi-structure fusion model to integrate different label s…
CoGeoAD: Hierarchical Color-Geometric Fusion with Multi-View Attention for Zero-Shot 3D Anomaly Detection
State Key Laboratory of Opto-Electronic Information Acquisition；Protection Technology, Anhui University, Hefei, China；School of Artificial Intelligence, Anhui University, Hefei, China；College of Intelligence Science；Technology, National University of Defense Technology, Changsha, China；School of Mathematics \& Physics, Anhui Jianzhu University, Hefei, China
With the development of technology, the usage areas and importance of biometric systems have started to increase. Since the characteristics of each person are different from each other, a single model biometric system can yield successful results. However, because the characteristics of twin people are very close to each other, multiple biometric systems including multiple characteristics of individuals will be more appropriate and will increase the recognition rate. In this study, a multiple biometric recognition system consisting of a combination of multiple algorithms and multiple models wa…
Real-world vision based applications require fine-grained classification for various area of interest like e-commerce, mobile applications, warehouse management, etc. where reducing the severity of mistakes and improving the classification accuracy is of utmost importance. This paper proposes a method to boost fine-grained classification through a hierarchical approach via learnable independent query embeddings. This is achieved through a classification network that uses coarse class predictions to improve the fine class accuracy in a stage-wise sequential manner. We exploit the idea of hierar…
Infrared and Visible Image Fusion with Hierarchical Human Perception
School of Electronic Engineering, Xidian University（西安电子科技大学电子工程学院）；Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications（重庆邮电大学图像认知重点实验室）
3D semantic segmentation is a fundamental building block for several scene understanding applications such as autonomous driving, robotics and AR/VR. Several state-of-the-art semantic segmentation models suffer from the part misclassification problem, wherein parts of the same object are labelled incorrectly. Previous methods have utilized hierarchical, iterative methods to fuse semantic and instance information, but they lack learnability in context fusion, and are computationally complex and heuristic driven. This paper presents Segment-Fusion, a novel attention-based method for hierarchical…
SenseTime Research（商汤科技研究院）；Beijing Institute of Technology（北京理工大学）
The growing availability of commodity RGB-D cameras has boosted the applications in the field of scene understanding. However, as a fundamental scene understanding task, surface normal estimation from RGB-D data lacks thorough investigation. In this paper, a hierarchical fusion network with adaptive feature re-weighting is proposed for surface normal estimation from a single RGB-D image. Specifically, the features from color image and depth are successively integrated at multiple scales to ensure global surface smoothness while preserving visually salient details. Meanwhile, the depth features…
Harbin Institute of Technology（哈尔滨工业大学）；Peng Cheng Laboratory（鹏城实验室）；Taobao (China) Software Co.,Ltd.（淘宝（中国）软件有限公司）；Tsinghua University（清华大学）
Depth map records distance between the viewpoint and objects in the scene, which plays a critical role in many real-world applications. However, depth map captured by consumer-grade RGB-D cameras suffers from low spatial resolution. Guided depth map super-resolution (DSR) is a popular approach to address this problem, which attempts to restore a high-resolution (HR) depth map from the input low-resolution (LR) depth and its coupled HR RGB image that serves as the guidance. The most challenging problems for guided DSR are how to correctly select consistent structures and propagate them, and pro…
Instituto Politécnico Nacional（国立理工学院）；National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）
Multimodal sentiment analysis is a very actively growing field of research. A promising area of opportunity in this field is to improve the multimodal fusion mechanism. We present a novel feature fusion strategy that proceeds in a hierarchical fashion, first fusing the modalities two in two and only then fusing all three modalities. On multimodal sentiment analysis of individual utterances, our strategy outperforms conventional concatenation of features by 1%, which amounts to 5% reduction in error rate. On utterance-level multimodal sentiment analysis of multi-utterance video clips, for which…
P$^2$HCT: Plug-and-Play Hierarchical C2F Transformer for Multi-Scale Feature Fusion
Department of Automation Tsinghua University Beijing, China；Department of R\&D Linsulabs Chengdu, China；School of Information；Communication Engineering University of Electronic Science
Little Strokes Fell Great Oaks: Boosting the Hierarchical Features for Multi-exposure Image Fusion
College of Computer Science and Technology, Zhejiang University of Technology（浙江工业大学计算机科学与技术学院）；School of Mechanical Engineering, Dalian University of Technology（大连理工大学机械工程学院）
A Cross-Hierarchical Difference Feature Fusion Network Based on Multiscale Encoder-Decoder for Hyperspectral Change Detection
School of Information and Communication Engineering, Hainan University（海南大学信息与通信工程学院）；Department of Artificial Intelligence, School of Information and Communication Engineering, Hainan University（海南大学信息与通信工程学院人工智能系）；Department of Management Information Systems, King Faisal University（国王法赫德大学管理信息系统系）
Bosch Center for Artificial Intelligence（博世人工智能中心）；University of Tübingen（图宾根大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）
Sensor fusion can significantly improve the performance of many computer vision tasks. However, traditional fusion approaches are either not data-driven and cannot exploit prior knowledge nor find regularities in a given dataset or they are restricted to a single application. We overcome this shortcoming by presenting a novel deep hierarchical variational autoencoder called FusionVAE that can serve as a basis for many fusion tasks. Our approach is able to generate diverse image samples that are conditioned on multiple noisy, occluded, or only partially visible input images. We derive and optim…
AdaMHF: Adaptive Multimodal Hierarchical Fusion for Survival Prediction
School of Computing and Information Technology, Great Bay University（大湾区大学计算与信息技术学院）；Harbin Institute of Technology（哈尔滨工业大学）；Beihang University（北京航空航天大学）；Macao Polytechnic University（澳门理工大学）；Dongguan Key Laboratory for Intelligence and Information Technology（东莞市智能与信息技术重点实验室）；Shenzhen Key Laboratory of Visual Object Detection and Recognition, Harbin Institute of Technology Shenzhen（哈尔滨工业大学深圳校区深圳市视觉目标检测与识别重点实验室）
Butter: Frequency Consistency and Hierarchical Fusion for Autonomous Driving Object Detection
Tsinghua University（清华大学）；University of Chinese Academy of Sciences（中国科学院大学）；Southeast University（东南大学）；University of Liverpool（利物浦大学）；Beijing Institute of Technology（北京理工大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
ARC Centre of Excellence for Robotic Vision（澳大利亚研究委员会机器人视觉卓越中心）；Queensland University of Technology（昆士兰科技大学）
Combining multiple complementary techniques together has long been regarded as a way to improve performance. In visual localization, multi-sensor fusion, multi-process fusion of a single sensing modality, and even combinations of different localization techniques have been shown to result in improved performance. However, merely fusing together different localization techniques does not account for the varying performance characteristics of different localization techniques. In this paper we present a novel, hierarchical localization system that explicitly benefits from three varying character…
Shanghai University（上海大学）；The University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）
Cloud detection plays a very important role in the process of remote sensing images. This paper designs a super-pixel level cloud detection method based on convolutional neural network (CNN) and deep forest. Firstly, remote sensing images are segmented into super-pixels through the combination of SLIC and SEEDS. Structured forests is carried out to compute edge probability of each pixel, based on which super-pixels are segmented more precisely. Segmented super-pixels compose a super-pixel level remote sensing database. Though cloud detection is essentially a binary classification problem, our …
Shenyang Aerospace University（沈阳航空航天大学）；Shenyang Fire Science and Technology Research Institute of MEM（沈阳消防科学研究所）
Rain streaks bring serious blurring and visual quality degradation, which often vary in size, direction and density. Current CNN-based methods achieve encouraging performance, while are limited to depict rain characteristics and recover image details in the poor visibility environment. To address these issues, we present a Multi-scale Hourglass Hierarchical Fusion Network (MH2F-Net) in end-to-end manner, to exactly captures rain streak features with multi-scale extraction, hierarchical distillation and information aggregation. For better extracting the features, a novel Multi-scale Hourglass E…
G2HFNet: GeoGran-Aware Hierarchical Feature Fusion Network for Salient Object Detection in Optical Remote Sensing Images
School of Control Science and Engineering, Shandong University（控制科学与工程学院，山东大学）；State Key Laboratory of Autonomous Intelligent Unmanned Systems（自主智能无人系统国家重点实验室）；School of Automation, Hangzhou Dianzi University（自动化学院，杭州电子科技大学）；School of information engineering, Huzhou University（信息工程学院，湖州大学）；School of Data Science, Lingnan University（数据科学学院，岭南大学）
Think Hierarchically, Act Dynamically: Hierarchical Multi-modal Fusion and Reasoning for Vision-and-Language Navigation
City University of Hong Kong, Dongguan Campus（香港城市大学东莞校区）；The University of Melbourne（墨尔本大学）；Tsinghua University（清华大学）；Baidu Inc.（百度公司）；University of Chinese Academy of Science（中国科学院大学）；National University of Singapore（新加坡国立大学）
HiPerformer: A High-Performance Global-Local Segmentation Model with Modular Hierarchical Fusion Strategy
Key Laboratory of Intelligent Computing and Signal Processing, Ministry of Education, Anhui University（智能计算与信号处理重点实验室，教育部，安徽大学）；Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education, East China University of Science and Technology（能源化工过程智能制造重点实验室，教育部，东华大学）
Hierarchical Modeling for Medical Visual Question Answering with Cross-Attention Fusion
School of Artificial Intelligence, Tiangong University（天津工业大学人工智能学院）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Linying Medical Technology (Shenzhen) Co., Ltd.（临影医疗科技（深圳）有限公司）
We present an approach named JSFusion (Joint Sequence Fusion) that can measure semantic similarity between any pairs of multimodal sequence data (e.g. a video clip and a language sentence). Our multimodal matching network consists of two key components. First, the Joint Semantic Tensor composes a dense pairwise representation of two sequence data into a 3D tensor. Then, the Convolutional Hierarchical Decoder computes their similarity score by discovering hidden hierarchical matches between the two sequence modalities. Both modules leverage hierarchical attention mechanisms that learn to promot…
CLFTv2: Efficient Camera-LiDAR Fusion for Semantic Segmentation via Hierarchical Feature Pyramids
Tallinn University of Technology（塔林理工大学）；FinEst Centre for Smart Cities, Tallinn University of Technology（塔林理工大学芬埃斯特智慧城市中心）；Universitas Mercatorum（梅尔卡托鲁姆大学）
Monocular depth estimation is a challenging task in complex compositions depicting multiple objects of diverse scales. Albeit the recent great progress thanks to the deep convolutional neural networks (CNNs), the state-of-the-art monocular depth estimation methods still fall short to handle such real-world challenging scenarios. In this paper, we propose a deep end-to-end learning framework to tackle these challenges, which learns the direct mapping from a color image to the corresponding depth map. First, we represent monocular depth estimation as a multi-category dense labeling task by contr…
DashFusion: Dual-stream Alignment with Hierarchical Bottleneck Fusion for Multimodal Sentiment Analysis
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Beijing National Research Center for Information Science and Technology, Tsinghua University（清华大学信息科学与技术国家研究中心）；Department of Automation, Tsinghua University（清华大学自动化系）
Robotic arm grasping is a fundamental operation in robotic control task goals. Most current methods for robotic grasping focus on RGB-D policy in the table surface scenario or 3D point cloud analysis and inference in the 3D space. Comparing to these methods, we propose a novel real-time multimodal hierarchical encoder-decoder neural network that fuses RGB and depth data to realize robotic humanoid grasping in 3D space with only partial observation. The quantification of raw depth data's uncertainty and depth estimation fusing RGB is considered. We develop a general labeling method to label gro…
School of Computer Science（计算机学院）；Northwestern Polytechnical University（西北工业大学）；Chinese Academy of Sciences（中国科学院）
Multiple modalities can provide more valuable information than single one by describing the same contents in various ways. Hence, it is highly expected to learn effective joint representation by fusing the features of different modalities. However, previous methods mainly focus on fusing the shallow features or high-level representations generated by unimodal deep networks, which only capture part of the hierarchical correlations across modalities. In this paper, we propose to densely integrate the representations by greedily stacking multiple shared layers between different modality-specific …
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；Megvii Research Nanjing, Megvii Technology（旷视科技南京研究院）
State-of-the-art two-stage object detectors apply a classifier to a sparse set of object proposals, relying on region-wise features extracted by RoIPool or RoIAlign as inputs. The region-wise features, in spite of aligning well with the proposal locations, may still lack the crucial context information which is necessary for filtering out noisy background detections, as well as recognizing objects possessing no distinctive appearances. To address this issue, we present a simple but effective Hierarchical Context Embedding (HCE) framework, which can be applied as a plug-and-play component, to f…
IT-TextFusion: Iterative Text-Image Interaction with Text-Guided Residual Refinement for Degradation-Aware Image Fusion
School of Automation, Southeast University（东南大学自动化学院）；Chair of Robotics, Artificial Intelligence and Real-time Systems, Technical University of Munich（慕尼黑工业大学机器人、人工智能与实时系统研究所）
HPFF: Hierarchical Locally Supervised Learning with Patch Feature Fusion
Southeast University（东南大学）；East China University of Science and Technology（华东理工大学）；University of Shanghai for Science and Technology（上海理工大学）；The Hong Kong University of Science and Technology（香港科技大学）；Nanyang Technological University（南洋理工大学）
