Local and global patterns of an object are closely related. Although each part of an object is incomplete, the underlying attributes about the object are shared among all parts, which makes reasoning the whole object from a single part possible. We hypothesize that a powerful representation of a 3D object should model the attributes that are shared between parts and the whole object, and distinguishable from other objects. Based on this hypothesis, we propose to learn point cloud representation by bidirectional reasoning between the local structures at different abstraction hierarchies and the global shape without human supervision. Experimental results on various benchmark datasets demonstrate the unsupervisedly learned representation is even better than supervised representation in discriminative power, generalization ability, and robustness. We show that unsupervisedly trained point cloud models can outperform their supervised counterparts on downstream classification tasks. Most notably, by simply increasing the channel width of an SSG PointNet++, our unsupervised model surpasses the state-of-the-art supervised methods on both synthetic and real-world 3D object classification datasets. We expect our observations to offer a new perspective on learning better representation from data structures instead of human annotations for point cloud understanding.
HKUST（香港科技大学）；UC San Diego（加州大学圣地亚哥分校）；Wuhan University（武汉大学）；Kuaishou Technology（快手科技）
We present radiance field propagation (RFP), a novel approach to segmenting objects in 3D during reconstruction given only unlabeled multi-view images of a scene. RFP is derived from emerging neural radiance field-based techniques, which jointly encodes semantics with appearance and geometry. The core of our method is a novel propagation strategy for individual objects' radiance fields with a bidirectional photometric loss, enabling an unsupervised partitioning of a scene into salient or meaningful regions corresponding to different object instances. To better handle complex scenes with multip…
In the era of end-to-end deep learning, many advances in computer vision are driven by large amounts of labeled data. In the optical flow setting, however, obtaining dense per-pixel ground truth for real scenes is difficult and thus such data is rare. Therefore, recent end-to-end convolutional networks for optical flow rely on synthetic datasets for supervision, but the domain mismatch between training and test scenarios continues to be a challenge. Inspired by classical energy-based optical flow methods, we design an unsupervised loss based on occlusion-aware bidirectional flow estimation and…
We propose DFPNet -- an unsupervised, joint learning system for monocular Depth, Optical Flow and egomotion (Camera Pose) estimation from monocular image sequences. Due to the nature of 3D scene geometry these three components are coupled. We leverage this fact to jointly train all the three components in an end-to-end manner. A single composite loss function -- which involves image reconstruction-based loss for depth & optical flow, bidirectional consistency checks and smoothness loss components -- is used to train the network. Using hyperparameter tuning, we are able to reduce the model size…
Tel Aviv University（特拉维夫大学）；Facebook AI Research（脸书人工智能研究院）
We study the problem of mapping between a domain $A$, in which there is a single training sample and a domain $B$, for which we have a richer training set. The method we present is able to perform this mapping in both directions. For example, we can transfer all MNIST images to the visual domain captured by a single SVHN image and transform the SVHN image to the domain of the MNIST images. Our method is based on employing one encoder and one decoder for each domain, without utilizing weight sharing. The autoencoder of the single sample domain is trained to match both this sample and the latent…
Shanghai Research Institute for Intelligent Autonomous Systems（上海智能自主系统研究院）；Shanghai Institute of Intelligent Science and Technology（上海智能科学技术研究院）；State Key Laboratory of Intelligent Autonomous Systems（智能自主系统国家重点实验室）；Frontiers Science Center for Intelligent Autonomous Systems（智能自主系统前沿科学中心）
Medical School of Nanjing University（南京大学医学院）；National Institute of Healthcare Data Science at Nanjing University（南京大学国家健康医疗大数据研究院）；State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；School of Computer Science and Technology, Nanjing University of Science and Technology（南京理工大学计算机科学与技术学院）；Department of Radiology, Nanjing Drum Tower Hospital, Nanjing University Medical School（南京大学医学院附属鼓楼医院放射科）；School of Biomedical Engineering, ShanghaiTech University（上海科技大学生物医学工程学院）；Department of Research and Development, Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能医疗科技有限公司研发部）；Department of Artificial Intelligence, Korea University（高丽大学人工智能系）
Accurate segmentation of brain tumors from multi-modal Magnetic Resonance (MR) images is essential in brain tumor diagnosis and treatment. However, due to the existence of domain shifts among different modalities, the performance of networks decreases dramatically when training on one modality and performing on another, e.g., train on T1 image while performing on T2 image, which is often required in clinical applications. This also prohibits a network from being trained on labeled data and then transferred to unlabeled data from a different domain. To overcome this, unsupervised domain adaptat…
CloDS: Visual-Only Unsupervised Cloth Dynamics Learning in Unknown Conditions
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学北京校区人工智能学院）；School of Engineering Science, University of Chinese Academy of Sciences（中国科学院大学工程科学学院）
University of Science and Technology of China（中国科学技术大学）；Microsoft Research Asia（微软亚洲研究院）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
Cross-modal correlation provides an inherent supervision for video unsupervised representation learning. Existing methods focus on distinguishing different video clips by visual and audio representations. We human visual perception could attend to regions where sounds are made, and our auditory perception could also ground their frequencies of sounding objects, which we call bidirectional local correspondence. Such supervision is intuitive but not well explored in the contrastive learning framework. This paper introduces a pretext task, Cross-Modal Attention Consistency (CMAC), for exploring t…
The Chinese University of Hong Kong（香港中文大学）；Imsight Medical Technology Co. Ltd.（视见医疗科技有限公司）；The Hong Kong Polytechnic University（香港理工大学）
Unsupervised domain adaptation has increasingly gained interest in medical image computing, aiming to tackle the performance degradation of deep neural networks when being deployed to unseen data with heterogeneous characteristics. In this work, we present a novel unsupervised domain adaptation framework, named as Synergistic Image and Feature Alignment (SIFA), to effectively adapt a segmentation network to an unlabeled target domain. Our proposed SIFA conducts synergistic alignment of domains from both image and feature perspectives. In particular, we simultaneously transform the appearance o…
Bosch Center for Artificial Intelligence（博世人工智能中心）；University of Bonn（波恩大学）
This paper introduces a novel method for self-supervised video representation learning via feature prediction. In contrast to the previous methods that focus on future feature prediction, we argue that a supervisory signal arising from unobserved past frames is complementary to one that originates from the future frames. The rationale behind our method is to encourage the network to explore the temporal structure of videos by distinguishing between future and past given present observations. We train our model in a contrastive learning framework, where joint encoding of future and past provide…
Tencent Jarvis Lab（腾讯贾维斯实验室）；National University of Defense Technology（国防科技大学）
Domain shift happens in cross-domain scenarios commonly because of the wide gaps between different domains: when applying a deep learning model well-trained in one domain to another target domain, the model usually performs poorly. To tackle this problem, unsupervised domain adaptation (UDA) techniques are proposed to bridge the gap between different domains, for the purpose of improving model performance without annotation in the target domain. Particularly, UDA has a great value for multimodal medical image analysis, where annotation difficulty is a practical concern. However, most existing …
The Chinese University of Hong Kong（香港中文大学）；University of Oxford（牛津大学）
2D image representations are in regular grids and can be processed efficiently, whereas 3D point clouds are unordered and scattered in 3D space. The information inside these two visual domains is well complementary, e.g., 2D images have fine-grained texture while 3D point clouds contain plentiful geometry information. However, most current visual recognition systems process them individually. In this paper, we present a \emph{bidirectional projection network (BPNet)} for joint 2D and 3D reasoning in an end-to-end manner. It contains 2D and 3D sub-networks with symmetric architectures, that are…
Multilingual Text-to-Image Person Retrieval via Bidirectional Relation Reasoning and Aligning
School of Computer Science and Technology, Soochow University（苏州大学计算机科学与技术学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）；Key Laboratory of New Generation Artificial Intelligence Technology & Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新 generation 人工智能技术及交叉应用重点实验室（东南大学），教育部，中国）
Visual reasoning with compositional natural language instructions, e.g., based on the newly-released Cornell Natural Language Visual Reasoning (NLVR) dataset, is a challenging task, where the model needs to have the ability to create an accurate mapping between the diverse phrases and the several objects placed in complex arrangements in the image. Further, this mapping needs to be processed to answer the question in the statement given the ordering and relationship of the objects across three similar images. In this paper, we propose a novel end-to-end neural model for the NLVR task, where we…
Sun Yat-sen University（中山大学）；DarkMatter AI Research（DarkMatter人工智能研究）
Recent researches on panoptic segmentation resort to a single end-to-end network to combine the tasks of instance segmentation and semantic segmentation. However, prior models only unified the two related tasks at the architectural level via a multi-branch scheme or revealed the underlying correlation between them by unidirectional feature fusion, which disregards the explicit semantic and co-occurrence relations among objects and background. Inspired by the fact that context information is critical to recognize and localize the objects, and inclusive object details are significant to parse th…
StabStitch++: Unsupervised Online Video Stitching with Spatiotemporal Bidirectional Warps
Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学学院）；Visual Intelligence +X International Cooperation Joint Laboratory of MOE, Beijing（教育部视觉智能+X国际合作联合实验室）；School of Computing and Data Science, Nanyang Technological University（南洋理工大学计算机与数据科学学院）；School of Media Engineering, Communication University of Zhejiang（浙江传媒学院媒体工程学院）；School of Information and Communication Engineering, University of Electronic Science and Technology of China（电子科技大学信息与通信工程学院）
Progressive Reasoning with Primitive Correction for Compositional Zero-Shot Learning
School of Computer Science and Technology, Beijing Jiaotong University（北京交通大学计算机科学与技术学院）；Key Laboratory of Big Data & Artificial Intelligence in Transportation (Ministry of Education)（交通运输大数据与人工智能教育部重点实验室）
The Chinese University of Hong Kong（香港中文大学）；T Stone Robotics Institute of CUHK（香港中文大学汤申机器人研究所）
Learning a good 3D human pose representation is important for human pose related tasks, e.g. human 3D pose estimation and action recognition. Within all these problems, preserving the intrinsic pose information and adapting to view variations are two critical issues. In this work, we propose a novel Siamese denoising autoencoder to learn a 3D pose representation by disentangling the pose-dependent and view-dependent feature from the human skeleton data, in a fully unsupervised manner. These two disentangled features are utilized together as the representation of the 3D pose. To consider both t…
Learning to estimate object pose often requires ground-truth (GT) labels, such as CAD model and absolute-scale object pose, which is expensive and laborious to obtain in the real world. To tackle this problem, we propose an unsupervised domain adaptation (UDA) for category-level object pose estimation, called UDA-COPE. Inspired by recent multi-modal UDA techniques, the proposed method exploits a teacher-student self-supervised learning scheme to train a pose estimation network without using target domain pose labels. We also introduce a bidirectional filtering method between the predicted norm…
University of Science and Technology of China（中国科学技术大学）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）；Tencent AI Lab（腾讯人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）
The advancement of visual tracking has continuously been brought by deep learning models. Typically, supervised learning is employed to train these models with expensive labeled data. In order to reduce the workload of manual annotations and learn to track arbitrary objects, we propose an unsupervised learning method for visual tracking. The motivation of our unsupervised learning is that a robust tracker should be effective in bidirectional tracking. Specifically, the tracker is able to forward localize a target object in successive frames and backtrace to its initial position in the first fr…
Raven's Progressive Matrices (RPM) is highly correlated with human intelligence, and it has been widely used to measure the abstract reasoning ability of humans. In this paper, to study the abstract reasoning capability of deep neural networks, we propose the first unsupervised learning method for solving RPM problems. Since the ground truth labels are not allowed, we design a pseudo target based on the prior constraints of the RPM formulation to approximate the ground truth label, which effectively converts the unsupervised learning strategy into a supervised one. However, the correct answer …
University of Toronto（多伦多大学）；The Hospital for Sick Children（SickKids医院（多伦多儿童医院））；Vector Institute（向量研究所）
Modern deep unsupervised learning methods have shown great promise for detecting diseases across a variety of medical imaging modalities. While previous generative modeling approaches successfully perform anomaly detection by learning the distribution of healthy 2D image slices, they process such slices independently and ignore the fact that they are correlated, all being sampled from a 3D volume. We show that incorporating the 3D context and processing whole-body MRI volumes is beneficial to distinguishing anomalies from their benign counterparts. In our work, we introduce a multi-channel sli…
Beyond Heatmaps: Unsupervised Concept-Graph Reasoning for Interpretable Visual Explanation
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Saarland University（萨尔大学）；Oldenburg University（奥尔登堡大学）；BEGO GmbH & Co. KG（BEGO有限两合公司）
Duke University（杜克大学）；Nanjing University（南京大学）；University at Buffalo（布法罗大学）
We investigate the non-identifiability issues associated with bidirectional adversarial training for joint distribution matching. Within a framework of conditional entropy, we propose both adversarial and non-adversarial approaches to learn desirable matched joint distributions for unsupervised and supervised tasks. We unify a broad family of adversarial models as joint distribution matching problems. Our approach stabilizes learning of unsupervised bidirectional adversarial learning methods. Further, we introduce an extension for semi-supervised learning tasks. Theoretical results are validat…
An unsupervised human action modeling framework can provide useful pose-sequence representation, which can be utilized in a variety of pose analysis applications. In this work we propose a novel temporal pose-sequence modeling framework, which can embed the dynamics of 3D human-skeleton joints to a continuous latent space in an efficient manner. In contrast to end-to-end framework explored by previous works, we disentangle the task of individual pose representation learning from the task of learning actions as a trajectory in pose embedding space. In order to realize a continuous pose embeddin…
Future Slot Prediction for Unsupervised Object Discovery in Surgical Video
PCASO Laboratory, Department of Surgery, University of Pennsylvania（匹兹堡大学外科实验室、外科系、匹兹堡大学）；Department of Computer and Information Science, University of Pennsylvania（计算机与信息科学系、匹兹堡大学）
With the growing popularity of short-form video sharing platforms such as \em{Instagram} and \em{Vine}, there has been an increasing need for techniques that automatically extract highlights from video. Whereas prior works have approached this problem with heuristic rules or supervised learning, we present an unsupervised learning approach that takes advantage of the abundance of user-edited videos on social media websites such as YouTube. Based on the idea that the most significant sub-events within a video class are commonly present among edited videos while less interesting ones appear less…
Warsaw University of Technology（华沙工业大学）；LIGM, Ecole des Ponts, Univ Gustave Eiffel（古斯塔夫·埃菲尔大学桥路学院LIGM实验室）；Tooploox（Tooploox公司）；IDEAS NCBR（IDEAS国家研究与发展中心）；Jagiellonian University of Cracow（克拉科夫雅盖隆大学）
Recent advances in visual representation learning allowed to build an abundance of powerful off-the-shelf features that are ready-to-use for numerous downstream tasks. This work aims to assess how well these features preserve information about the objects, such as their spatial location, their visual properties and their relative relationships. We propose to do so by evaluating them in the context of visual reasoning, where multiple objects with complex relationships and different attributes are at play. More specifically, we introduce a protocol to evaluate visual representations for the task…
Despite the significant progress that has been made on estimating optical flow recently, most estimation methods, including classical and deep learning approaches, still have difficulty with multi-scale estimation, real-time computation, and/or occlusion reasoning. In this paper, we introduce dilated convolution and occlusion reasoning into unsupervised optical flow estimation to address these issues. The dilated convolution allows our network to avoid upsampling via deconvolution and the resulting gridding artifacts. Dilated convolution also results in a smaller memory footprint which speeds …
Stabilizing Unsupervised Self-Evolution of MLLMs via Continuous Softened Retracing reSampling
Tsinghua University（清华大学）；Hefei University of Technology（合肥工业大学）；University of Arizona（亚利桑那大学）；MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统实验室）
Existing unsupervised video-to-video translation methods fail to produce translated videos which are frame-wise realistic, semantic information preserving and video-level consistent. In this work, we propose UVIT, a novel unsupervised video-to-video translation model. Our model decomposes the style and the content, uses the specialized encoder-decoder structure and propagates the inter-frame information through bidirectional recurrent neural network (RNN) units. The style-content decomposition mechanism enables us to achieve style consistent video translation results as well as provides us wit…
