In this paper, we present the PS^2-Net -- a locally and globally aware deep learning framework for semantic segmentation on 3D scene-level point clouds. In order to deeply incorporate local structures and global context to support 3D scene segmentation, our network is built on four repeatedly stacked encoders, where each encoder has two basic components: EdgeConv that captures local structures and NetVLAD that models global context. Different from existing start-of-the-art methods for point-based scene semantic segmentation that either violate or do not achieve permutation invariance, our PS^2-Net is designed to be permutation invariant which is an essential property of any deep network used to process unordered point clouds. We further provide theoretical proof to guarantee the permutation invariance property of our network. We perform extensive experiments on two large-scale 3D indoor scene datasets and demonstrate that our PS2-Net is able to achieve state-of-the-art performances as compared to existing approaches.
Hanoi University of Science and Technology（河内科技大学）；MICA International Research Institute（MICA国际研究院）；Asilla Inc.（Asilla公司）
3D hand pose estimation from RGB images suffers from the difficulty of obtaining the depth information. Therefore, a great deal of attention has been spent on estimating 3D hand pose from 2D hand joints. In this paper, we leverage the advantage of spatial-temporal Graph Convolutional Neural Networks and propose LG-Hand, a powerful method for 3D hand pose estimation. Our method incorporates both spatial and temporal dependencies into a single process. We argue that kinematic information plays an important role, contributing to the performance of 3D hand pose estimation. We thereby introduce two…
We propose a novel feed-forward network for video inpainting. We use a set of sampled video frames as the reference to take visible contents to fill the hole of a target frame. Our video inpainting network consists of two stages. The first stage is an alignment module that uses computed homographies between the reference frames and the target frame. The visible patches are then aggregated based on the frame similarity to fill in the target holes roughly. The second stage is a non-local attention module that matches the generated patches with known reference patches (in space and time) to refin…
State-of-the-art spoof detection methods tend to overfit to the spoof types seen during training and fail to generalize to unknown spoof types. Given that face anti-spoofing is inherently a local task, we propose a face anti-spoofing framework, namely Self-Supervised Regional Fully Convolutional Network (SSR-FCN), that is trained to learn local discriminative cues from a face image in a self-supervised manner. The proposed framework improves generalizability while maintaining the computational efficiency of holistic face anti-spoofing approaches (< 4 ms on a Nvidia GTX 1080Ti GPU). The propose…
South China University of Technology（华南理工大学）；Peng Cheng Laboratory（鹏城实验室）
The challenges of learning a robust 6D pose function lie in 1) severe occlusion and 2) systematic noises in depth images. Inspired by the success of point-pair features, the goal of this paper is to recover the 6D pose of an object instance segmented from RGB-D images by locally matching pairs of oriented points between the model and camera space. To this end, we propose a novel Bi-directional Correspondence Mapping Network (BiCo-Net) to first generate point clouds guided by a typical pose regression, which can thus incorporate pose-sensitive information to optimize generation of local coordin…
Fudan University（复旦大学）；School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；Shanghai Key Lab of Intelligent Information Processing（上海市智能信息处理重点实验室）
Face aging is of great importance for cross-age recognition and entertainment-related applications. Recently, conditional generative adversarial networks (cGANs) have achieved impressive results for face aging. Existing cGANs-based methods usually require a pixel-wise loss to keep the identity and background consistent. However, minimizing the pixel-wise loss between the input and synthesized images likely resulting in a ghosted or blurry face. To address this deficiency, this paper introduces an Attention Conditional GANs (AcGANs) approach for face aging, which utilizes attention mechanism to…
Technische Universität München（慕尼黑工业大学）；Konica Minolta Laboratory Europe（柯尼卡美能达欧洲实验室）；Johns Hopkins University（约翰斯·霍普金斯大学）；Trinity College Dublin（都柏林圣三一学院）
Registration of partial-view 3D US volumes with MRI data is influenced by initialization. The standard of practice is using extrinsic or intrinsic landmarks, which can be very tedious to obtain. To overcome the limitations of registration initialization, we present a novel approach that is based on Euclidean distance maps derived from easily obtainable coarse segmentations. We evaluate our approach quantitatively on the publicly available RESECT dataset and show that it is robust regarding overlap of target area and initial position. Furthermore, our method provides initializations that are su…
University of California, Santa Barbara（加州大学圣巴巴拉分校）
A major challenge for video captioning is to combine audio and visual cues. Existing multi-modal fusion methods have shown encouraging results in video understanding. However, the temporal structures of multiple modalities at different granularities are rarely explored, and how to selectively fuse the multi-modal representations at different levels of details remains uncharted. In this paper, we propose a novel hierarchically aligned cross-modal attention (HACA) framework to learn and selectively fuse both global and local temporal dynamics of different modalities. Furthermore, for the first t…
Stanford University（斯坦福大学）；University of Cambridge（剑桥大学）；ENGIE Lab CRIGEN（ENGIE集团CRIGEN实验室）；Dubai Energy and Water Authority（迪拜水电局）
Solar forecasting from ground-based sky images has shown great promise in reducing the uncertainty in solar power generation. With more and more sky image datasets open sourced in recent years, the development of accurate and reliable deep learning-based solar forecasting methods has seen a huge growth in potential. In this study, we explore three different training strategies for solar forecasting models by leveraging three heterogeneous datasets collected globally with different climate patterns. Specifically, we compare the performance of local models trained individually based on single da…
LIX, École Polytechnique, IP Paris（巴黎理工学院综合理工学院LIX实验室）
In this work, we present a novel learning-based framework that combines the local accuracy of contrastive learning with the global consistency of geometric approaches, for robust non-rigid matching. We first observe that while contrastive learning can lead to powerful point-wise features, the learned correspondences commonly lack smoothness and consistency, owing to the purely combinatorial nature of the standard contrastive losses. To overcome this limitation we propose to boost contrastive feature learning with two types of smoothness regularization that inject geometric information into cor…
Apple Inc.（苹果公司）；Carnegie Mellon University（卡内基梅隆大学）
Registration of 3D point clouds is a fundamental task in several applications of robotics and computer vision. While registration methods such as iterative closest point and variants are very popular, they are only locally optimal. There has been some recent work on globally optimal registration, but they perform poorly in the presence of noise in the measurements. In this work we develop a mixed integer programming-based approach for globally optimal registration that explicitly considers uncertainty in its optimization, and hence produces more accurate estimates. Furthermore, from a practica…
Everywhere Attack: Attacking Locally and Globally to Boost Targeted Transferability
Southwest university of science and technology（西南科技大学）；Guangan institute of technology（广安职业技术学院）；City University of Macau（澳门城市大学）；Beijing normal university（北京师范大学）
Learning Locally, Revising Globally: Global Reviser for Federated Learning with Noisy Labels
College of Computer Science, Sichuan University, Chengdu, China（四川大学计算机学院，中国成都）；Engineering Research Center of Machine Learning（机器学习工程研究中心）；University of Sydney, Sydney, Australia（悉尼大学，澳大利亚悉尼）；Southeast University, Nanjing, China（东南大学，中国南京）
Seeing Globally, Refining Locally: Global Visual Guidance and Local Ultrasound Cues for Robust Freehand 3-D Ultrasound Reconstruction
Department of Mechanical Engineering, The University of Hong Kong（香港大学机械工程系）；Multi-Scale Medical Robotics Center（多尺度医疗机器人中心）；Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong（香港中文大学机械与自动化工程系）
Attri-Net: A Globally and Locally Inherently Interpretable Model for Multi-Label Classification Using Class-Specific Counterfactuals
Cluster of Excellence: Machine Learning - New Perspectives for Science, University of Tübingen（机器学习新视角科学卓越中心，图宾根大学）；Department of Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg（计算机科学系，埃尔兰根-纽伦堡弗里德里希-亚历山大大学）；Hertie Institute for Artificial Intelligence in Brain Health, University of Tübingen（脑健康人工智能研究所，图宾根大学）；Department of Diabetes, Endocrinology, Nutritional Medicine and Metabolism, Inselspital, Bern University Hospital, University of Bern（糖尿病、内分泌学、营养医学和代谢学系，伯恩大学医院，伯恩大学）；Faculty of Health Sciences and Medicine, University of Lucerne（健康科学与医学学院，卢塞恩大学）
