Since humans interact with diverse objects every day, the holistic 3D capture of these interactions is important to understand and model human behaviour. However, most existing methods for hand-object reconstruction from RGB either assume pre-scanned object templates or heavily rely on limited 3D hand-object data, restricting their ability to scale and generalize to more unconstrained interaction settings. To this end, we introduce HOLD -- the first category-agnostic method that reconstructs an articulated hand and object jointly from a monocular interaction video. We develop a compositional articulated implicit model that can reconstruct disentangled 3D hand and object from 2D images. We also further incorporate hand-object constraints to improve hand-object poses and consequently the reconstruction quality. Our method does not rely on 3D hand-object annotations while outperforming fully-supervised baselines in both in-the-lab and challenging in-the-wild settings. Moreover, we qualitatively show its robustness in reconstructing from in-the-wild videos. Code: https://github.com/zc-alexfan/hold
Politecnico di Torino（都灵理工大学）；Italian Institute of Technology（意大利理工学院）
In this paper, we present a deep learning model that exploits the power of self-supervision to perform 3D point cloud completion, estimating the missing part and a context region around it. Local and global information are encoded in a combined embedding. A denoising pretext task provides the network with the needed local cues, decoupled from the high-level semantics and naturally shared over multiple classes. On the other hand, contrastive learning maximizes the agreement between variants of the same shape with different missing portions, thus producing a representation which captures the glo…
TrackAny3D: Transferring Pretrained 3D Models for Category-unified 3D Point Cloud Tracking
Zhejiang University of Technology（浙江工业大学）；RMIT University（皇家墨尔本理工大学）；Zhejiang Key Laboratory of Visual Information Intelligent Processing（浙江省视觉信息智能处理重点实验室）
Recent work on single-view 3D reconstruction shows impressive results, but has been restricted to a few fixed categories where extensive training data is available. The problem of generalizing these models to new classes with limited training data is largely open. To address this problem, we present a new model architecture that reframes single-view 3D reconstruction as learnt, category agnostic refinement of a provided, category-specific prior. The provided prior shape for a novel class can be obtained from as few as one 3D shape from this class. Our model can start reconstructing objects fro…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Snap Inc.（Snap公司）
Semantic keypoints provide concise abstractions for a variety of visual understanding tasks. Existing methods define semantic keypoints separately for each category with a fixed number of semantic labels in fixed indices. As a result, this keypoint representation is in-feasible when objects have a varying number of parts, e.g. chairs with varying number of legs. We propose a category-agnostic keypoint representation, which combines a multi-peak heatmap (StarMap) for all the keypoints and their corresponding features as 3D locations in the canonical viewpoint (CanViewFeature) defined for each i…
JD AI Research（京东人工智能研究院）；City University of Hong Kong（香港城市大学）
Unsupervised domain adaptation has received significant attention in recent years. Most of existing works tackle the closed-set scenario, assuming that the source and target domains share the exactly same categories. In practice, nevertheless, a target domain often contains samples of classes unseen in source domain (i.e., unknown class). The extension of domain adaptation from closed-set to such open-set situation is not trivial since the target samples in unknown class are not expected to align with the source. In this paper, we address this problem by augmenting the state-of-the-art domain …
Visual Computing Institute, RWTH Aachen University（亚琛工业大学视觉计算研究所）
The most common paradigm for vision-based multi-object tracking is tracking-by-detection, due to the availability of reliable detectors for several important object categories such as cars and pedestrians. However, future mobile systems will need a capability to cope with rich human-made environments, in which obtaining detectors for every possible object category would be infeasible. In this paper, we propose a model-free multi-object tracking approach that uses a category-agnostic image segmentation method to track objects. We present an efficient segmentation mask-based tracker which associ…
The Chinese University of Hong Kong（香港中文大学）；SenseTime Research（商汤科技研究院）；The University of Hong Kong（香港大学）；Shanghai AI Laboratory（上海人工智能实验室）；The University of Sydney（悉尼大学）
Existing works on 2D pose estimation mainly focus on a certain category, e.g. human, animal, and vehicle. However, there are lots of application scenarios that require detecting the poses/keypoints of the unseen class of objects. In this paper, we introduce the task of Category-Agnostic Pose Estimation (CAPE), which aims to create a pose estimation model capable of detecting the pose of any class of object given only a few samples with keypoint definition. To achieve this goal, we formulate the pose estimation problem as a keypoint matching problem and design a novel CAPE framework, termed POs…
Few-Shot Anomaly Detection via Category-Agnostic Registration Learning
College of Information Communications, National University of Defense Technology（国防科技大学信息通信学院）；Cooperative Medianet Innovation Center, Shanghai Jiao Tong University（上海交通大学协同媒体网络创新中心）；National University of Singapore（新加坡国立大学）；King’s College London（伦敦国王学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
We tackle the task of Class Agnostic Counting, which aims to count objects in a novel object category at test time without any access to labeled training data for that category. All previous class agnostic counting methods cannot work in a fully automated setting, and require computationally expensive test time adaptation. To address these challenges, we propose a visual counter which operates in a fully automated setting and does not require any test time adaptation. Our proposed approach first identifies exemplars from repeating objects in an image, and then counts the repeating objects. We …
Towards Distribution-Agnostic Generalized Category Discovery
Zhejiang University（浙江大学）；The Hong Kong University of Science and Technology（香港科技大学）；Centre for Frontier AI Research, A*STAR（新加坡科技研究局前沿人工智能研究中心）；Angelalign Technology Inc.（时代天使科技有限公司）
Stanford University（斯坦福大学）；Google Inc.（谷歌公司）；Facebook AI Research（脸书人工智能研究院）
We investigate the problem of learning category-specific 3D shape reconstruction from a variable number of RGB views of previously unobserved object instances. Most approaches for multiview shape reconstruction operate on sparse shape representations, or assume a fixed number of views. We present a method that can estimate dense 3D shape, and aggregate shape across multiple and varying number of input views. Given a single input view of an object instance, we propose a representation that encodes the dense shape of the visible object surface as well as the surface behind line of sight occluded…
CNS-Edit++: Category-Agnostic 3D Editing with Coupled Neural Shape Representation
The Chinese University of Hong Kong（香港中文大学）；Lingnan University（岭南大学）；National University of Singapore（新加坡国立大学）；The Hong Kong University of Science and Technology（香港科技大学）；Autodesk AI Lab（欧特克人工智能实验室）；Simon Fraser University（西蒙弗雷泽大学）
University of Science and Technology of China（中国科学技术大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Recovering the skeletal shape of an animal from a monocular video is a longstanding challenge. Prevailing animal reconstruction methods often adopt a control-point driven animation model and optimize bone transforms individually without considering skeletal topology, yielding unsatisfactory shape and articulation. In contrast, humans can easily infer the articulation structure of an unknown animal by associating it with a seen articulated character in their memory. Inspired by this fact, we present CASA, a novel Category-Agnostic Skeletal Animal reconstruction method consisting of two major co…
Adversarial Reconstruction Feedback for Robust Fine-grained Generalization
College of Computer and Engineering, Shandong University of Science and Technology（山东科技大学计算机与工程学院）；School of Software, Dalian University of Technology（大连理工大学软件学院）
GCR: Geometry-Consistent Routing for Task-Agnostic Continual Anomaly Detection
Tsinghua University Shenzhen International Graduate School（清华大学深圳国际研究生院）；Ratel Soft；Affiliated Fifth Hospital, Wenzhou Medical University（温州医学院附属第五医院）；Chinese Medicine Guangdong Laboratory（广东中医药实验室）
We present a neural network approach to transfer the motion from a single image of an articulated object to a rest-state (i.e., unarticulated) 3D model. Our network learns to predict the object's pose, part segmentation, and corresponding motion parameters to reproduce the articulation shown in the input image. The network is composed of three distinct branches that take a shared joint image-shape embedding and is trained end-to-end. Unlike previous methods, our approach is independent of the topology of the object and can work with objects from arbitrary categories. Our method, trained with o…
Universidad de Zaragoza（萨拉戈萨大学）；Bitbrain Technologies（比特脑科技）
Hand action recognition is a special case of action recognition with applications in human-robot interaction, virtual reality or life-logging systems. Building action classifiers able to work for such heterogeneous action domains is very challenging. There are very subtle changes across different actions from a given application but also large variations across domains (e.g. virtual reality vs life-logging). This work introduces a novel skeleton-based hand motion representation model that tackles this problem. The framework we propose is agnostic to the application domain or camera recording v…
Does it Really Count? Assessing Semantic Grounding in Text-Guided Class-Agnostic Counting
Institute of Information Science and Technologies of the National Research Council (ISTI-CNR)（意大利国家研究理事会信息科学与技术研究所）；University of Pisa - Department of Information Engineering（比萨大学信息工程系）
Johns Hopkins University（约翰斯·霍普金斯大学）；Mercedes-Benz Research and Development India（梅赛德斯-奔驰印度研发中心）
Existing approaches for unsupervised domain adaptive object detection perform feature alignment via adversarial training. While these methods achieve reasonable improvements in performance, they typically perform category-agnostic domain alignment, thereby resulting in negative transfer of features. To overcome this issue, in this work, we attempt to incorporate category information into the domain adaptation process by proposing Memory Guided Attention for Category-Aware Domain Adaptation (MeGA-CDA). The proposed method consists of employing category-wise discriminators to ensure category-awa…
Object tracking can be formulated as "finding the right object in a video". We observe that recent approaches for class-agnostic tracking tend to focus on the "finding" part, but largely overlook the "object" part of the task, essentially doing a template matching over a frame in a sliding-window. In contrast, class-specific trackers heavily rely on object priors in the form of category-specific object detectors. In this work, we re-purpose category-specific appearance models into a generic objectness prior. Our approach converts a category-specific object detector into a category-agnostic, ob…
In this work, we focus on a novel task of category-level functional hand-object manipulation synthesis covering both rigid and articulated object categories. Given an object geometry, an initial human hand pose as well as a sparse control sequence of object poses, our goal is to generate a physically reasonable hand-object manipulation sequence that performs like human beings. To address such a challenge, we first design CAnonicalized Manipulation Spaces (CAMS), a two-level space hierarchy that canonicalizes the hand poses in an object-centric and contact-centric view. Benefiting from the repr…
Estimating the 6D pose and size of household containers is challenging due to large intra-class variations in the object properties, such as shape, size, appearance, and transparency. The task is made more difficult when these objects are held and manipulated by a person due to varying degrees of hand occlusions caused by the type of grasps and by the viewpoint of the camera observing the person holding the object. In this paper, we present a mixed-reality dataset of hand-occluded containers for category-level 6D object pose and size estimation. The dataset consists of 138,240 images of render…
CASA: Class-Agnostic Shared Attributes in Vision-Language Models for Efficient Incremental Object Detection
School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）；Peng Cheng Laboratory（鹏城实验室）；School of Computer Science, Peking University（北京大学计算机学院）
