The majority of descriptor-based methods for geometric processing of non-rigid shape rely on hand-crafted descriptors. Recently, learning-based techniques have been shown effective, achieving state-of-the-art results in a variety of tasks. Yet, even though these methods can in principle work directly on raw data, most methods still rely on hand-crafted descriptors at the input layer. In this work, we wish to challenge this practice and use a neural network to learn descriptors directly from the raw mesh. To this end, we introduce two modules into our neural architecture. The first is a local reference frame (LRF) used to explicitly make the features invariant to rigid transformations. The second is continuous convolution kernels that provide robustness to sampling. We show the efficacy of our proposed network in learning on raw meshes using two cornerstone tasks: shape matching, and human body parts segmentation. Our results show superior results over baseline methods that use hand-crafted descriptors.
Feature descriptors play a crucial role in a wide range of geometry analysis and processing applications, including shape correspondence, retrieval, and segmentation. In this paper, we introduce Geodesic Convolutional Neural Networks (GCNN), a generalization of the convolutional networks (CNN) paradigm to non-Euclidean manifolds. Our construction is based on a local geodesic system of polar coordinates to extract "patches", which are then passed through a cascade of filters and linear and non-linear operators. The coefficients of the filters and linear combination weights are optimization vari…
We propose DualConvMesh-Nets (DCM-Net) a family of deep hierarchical convolutional networks over 3D geometric data that combines two types of convolutions. The first type, geodesic convolutions, defines the kernel weights over mesh surfaces or graphs. That is, the convolutional kernel weights are mapped to the local surface of a given mesh. The second type, Euclidean convolutions, is independent of any underlying mesh structure. The convolutional kernel is applied on a neighborhood obtained from a local affinity representation based on the Euclidean distance between 3D points. Intuitively, geo…
Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院））；Shandong Artificial Intelligence Institute（山东人工智能研究院）；ENS Paris-Saclay（巴黎萨克雷高等师范学院）；CNRS（法国国家科学研究中心）；University Paris-Saclay（巴黎萨克雷大学）；Hong Kong Centre for Cerebro-cardiovascular Health Engineering（香港心脑血管健康工程研究中心）；University Paris Dauphine（巴黎多芬大学）；PSL Research University（PSL研究大学）
The minimal geodesic models based on the Eikonal equations are capable of finding suitable solutions in various image segmentation scenarios. Existing geodesic-based segmentation approaches usually exploit image features in conjunction with geometric regularization terms, such as Euclidean curve length or curvature-penalized length, for computing geodesic curves. In this paper, we take into account a more complicated problem: finding curvature-penalized geodesic paths with a convexity shape prior. We establish new geodesic models relying on the strategy of orientation-lifting, by which a plana…
FAAGC: Feature Augmentation on Adaptive Geodesic Curve Based on the shape space theory
School of Computer Engineering and Science, Shanghai University（上海大学计算机工程与科学学院）；Key Laboratory of Silicate Cultural Relics Conservation (Shanghai University), Ministry of Education（上海大学硅酸盐文物保护重点实验室（教育部））
Learning Geodesics of Geometric Shape Deformations From Images
University of Virginia, Department of Electrical & Computer Engineering, Charlottesville, VA, USA（弗吉尼亚大学电气与计算机工程系）；University of Virginia, Department of Computer Science, Charlottesville, VA, USA（弗吉尼亚大学计算机科学系）
Technion（以色列理工学院）；Tel Aviv University（特拉维夫大学）；Università della Svizzera Italiana（瑞士意大利语区大学）
Natural objects can be subject to various transformations yet still preserve properties that we refer to as invariants. Here, we use definitions of affine invariant arclength for surfaces in R^3 in order to extend the set of existing non-rigid shape analysis tools. In fact, we show that by re-defining the surface metric as its equi-affine version, the surface with its modified metric tensor can be treated as a canonical Euclidean object on which most classical Euclidean processing and analysis tools can be applied. The new definition of a metric is used to extend the fast marching method techn…
Imperial College London（帝国理工学院）；OCM Digital Media（OCM数字媒体）
Geodesic paths and distances are among the most popular intrinsic properties of 3D surfaces. Traditionally, geodesic paths on discrete polygon surfaces were computed using shortest path algorithms, such as Dijkstra. However, such algorithms have two major limitations. They are non-differentiable which limits their direct usage in learnable pipelines and they are considerably time demanding. To address such limitations and alleviate the computational burden, we propose a learnable network to approximate geodesic paths. The proposed method is comprised by three major components: a graph neural n…
This work addresses the problem of non-rigid registration of 3D scans, which is at the core of shape modeling techniques. Firstly, we propose a new kernel based on geodesic distances for the Gaussian Process Morphable Models (GPMMs) framework. The use of geodesic distances into the kernel makes it more adapted to the topological and geometric characteristics of the surface and leads to more realistic deformations around holes and curved areas. Since the kernel possesses hyperparameters we have optimized them for the task of face registration on the FaceWarehouse dataset. We show that the Geode…
This paper addresses 3D shape recognition. Recent work typically represents a 3D shape as a set of binary variables corresponding to 3D voxels of a uniform 3D grid centered on the shape, and resorts to deep convolutional neural networks(CNNs) for modeling these binary variables. Robust learning of such CNNs is currently limited by the small datasets of 3D shapes available, an order of magnitude smaller than other common datasets in computer vision. Related work typically deals with the small training datasets using a number of ad hoc, hand-tuning strategies. To address this issue, we formulate…
In the elastic shape analysis approach to shape matching and object classification, plane curves are represented as points in an infinite-dimensional Riemannian manifold, wherein shape dissimilarity is measured by geodesic distance. A remarkable result of Younes, Michor, Shah and Mumford says that the space of closed planar shapes, endowed with a natural metric, is isometric to an infinite-dimensional Grassmann manifold via the so-called square root transform. This result facilitates efficient shape comparison by virtue of explicit descriptions of Grassmannian geodesics. In this paper, we exte…
Robarts Research Institute（罗伯茨研究所）；Western University（韦仕敦大学）
Although topological considerations amongst multiple labels have been previously investigated in the context of continuous max-flow image segmentation, similar investigations have yet to be made about shape considerations in a general and extendable manner. This paper presents shape complexes for segmentation, which capture more complex shapes by combining multiple labels and super-labels constrained by geodesic star convexity. Shape complexes combine geodesic star convexity constraints with hierarchical label organization, which together allow for more complex shapes to be represented. This f…
In recent years, deep learning has achieved great success in many computer vision applications. Convolutional neural networks (CNNs) have lately emerged as a major approach to image classification. Most research on CNNs thus far has focused on developing architectures such as the Inception and residual networks. The convolution layer is the core of the CNN, but few studies have addressed the convolution unit itself. In this paper, we introduce a convolution unit called the active convolution unit (ACU). A new convolution has no fixed shape, because of which we can define any form of convolutio…
Aix Marseille Univ（艾克斯-马赛大学）；Université de Toulon（土伦大学）；CNRS（法国国家科学研究中心）；LIS（信息科学实验室）
In this paper, we propose a method for characterizing 3D shapes from point clouds and we show a direct application on a study of organ temporal deformations. As an example, we characterize the behavior of a bladder during a forced respiratory motion with a reduced number of 3D surface points: first, a set of equidistant points representing the vertices of quadrilateral mesh for the surface in the first time frame are tracked throughout a long dynamic MRI sequence using a Large Deformation Diffeomorphic Metric Mapping (LDDMM) framework. Second, a novel geometric feature which is invariant to sc…
Geodesics with Unified Tangent-constrained Priors and Curvature Regularization
Shandong Artificial Intelligence Institute, Qilu University of Technology (Shandong Academy of Sciences)（山东省人工智能研究院，齐鲁工业大学（山东省科学院））；Yuanshen Rehabilitation Institute, Shanghai Jiao Tong University School of Medicine（元身康复研究院，上海交通大学医学院）；School of Control Science and Engineering, Shandong University（控制科学与工程学院，山东大学）；Department of Radiation Oncology, Shandong Cancer Hospital and Institute, Shandong First Medical University, Shandong Academy of Medical Sciences（放疗科，山东省肿瘤医院及研究院，山东第一医科大学，山东省医学科学院）；CEREMADE, Université Paris Dauphine, Université-PSL, CNRS, UMR 7534（CEREMADE，巴黎大学Dauphine，Université-PSL，CNRS，UMR 7534）
Auditing Training-Free 3D Shape Retrieval with Diffused Geodesic Moments
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学）；Guangzhou International Economics College（广州国际经济学院）；School of Electrical and Electronic Engineering, The University of Sheffield（谢菲尔德大学电子与电气工程学院）
Convolutional neural networks (CNNs) have massively impacted visual recognition in 2D images, and are now ubiquitous in state-of-the-art approaches. CNNs do not easily extend, however, to data that are not represented by regular grids, such as 3D shape meshes or other graph-structured data, to which traditional local convolution operators do not directly apply. To address this problem, we propose a novel graph-convolution operator to establish correspondences between filter weights and graph neighborhoods with arbitrary connectivity. The key novelty of our approach is that these correspondence…
University of Sydney（悉尼大学）；University of New South Wales（新南威尔士大学）
Point cloud is a principal data structure adopted for 3D geometric information encoding. Unlike other conventional visual data, such as images and videos, these irregular points describe the complex shape features of 3D objects, which makes shape feature learning an essential component of point cloud analysis. To this end, a shape-oriented message passing scheme dubbed ShapeConv is proposed to focus on the representation learning of the underlying shape formed by each local neighboring point. Despite this intra-shape relationship learning, ShapeConv is also designed to incorporate the contextu…
Knowledge of shape geometry plays a pivotal role in many shape analysis applications. In this paper we introduce a local geometry-inclusive global representation of 3D shapes based on computation of the shortest quasi-geodesic paths between all possible pairs of points on the 3D shape manifold. In the proposed representation, the normal curvature along the quasi-geodesic paths between any two points on the shape surface is preserved. We employ the eigenspectrum of the proposed global representation to address the problems of determination of region-based correspondence between isometric shapes…
This paper proposes a novel deep learning approach for single image defocus deblurring based on inverse kernels. In a defocused image, the blur shapes are similar among pixels although the blur sizes can spatially vary. To utilize the property with inverse kernels, we exploit the observation that when only the size of a defocus blur changes while keeping the shape, the shape of the corresponding inverse kernel remains the same and only the scale changes. Based on the observation, we propose a kernel-sharing parallel atrous convolutional (KPAC) block specifically designed by incorporating the p…
Three-dimensional medical imaging enables detailed understanding of osteoarthritis structural status. However, there remains a vast need for automatic, thus, reader-independent measures that provide reliable assessment of subject-specific clinical outcomes. To this end, we derive a consistent generalization of the recently proposed B-score to Riemannian shape spaces. We further present an algorithmic treatment yielding simple, yet efficient computations allowing for analysis of large shape populations with several thousand samples. Our intrinsic formulation exhibits improved discrimination abi…
University of Utah（犹他大学）；Florida State University（佛罗里达州立大学）；University of Vienna（维也纳大学）；University of Virginia（弗吉尼亚大学）
The goal of diffusion-weighted magnetic resonance imaging (DWI) is to infer the structural connectivity of an individual subject's brain in vivo. To statistically study the variability and differences between normal and abnormal brain connectomes, a mathematical model of the neural connections is required. In this paper, we represent the brain connectome as a Riemannian manifold, which allows us to model neural connections as geodesics. This leads to the challenging problem of estimating a Riemannian metric that is compatible with the DWI data, i.e., a metric such that the geodesic curves repr…
In this paper, we aim to reconstruct a full 3D human shape from a single image. Previous vertex-level and parameter regression approaches reconstruct 3D human shape based on a pre-defined adjacency matrix to encode positive relations between nodes. The deep topological relations for the surface of the 3D human body are not carefully exploited. Moreover, the performance of most existing approaches often suffer from domain gap when handling more occlusion cases in real-world scenes. In this work, we propose a Deep Mesh Relation Capturing Graph Convolution Network, DC-GNet, with a shape completio…
Computer vision is a growing field with a lot of new applications in automation and robotics, since it allows the analysis of images and shapes for the generation of numerical or analytical information. One of the most used method of information extraction is image filtering through convolution kernels, with each kernel specialized for specific applications. The objective of this paper is to present a novel convolution kernels, based on principles of electromagnetic potentials and fields, for a general use in computer vision and to demonstrate its usage for shape and stroke analysis. Such filt…
LDConv: Linear deformable convolution for improving convolutional neural networks
College of Computer and Information Science, Chongqing Normal University（重庆师范大学计算机与信息科学学院）；Chongqing Engineering Research Center of Educational Big Data Intelligent Perception and Application（重庆教育大数据智能感知与应用工程研究中心）；College of Electronic and Information Engineering, Southwest University（西南大学电子信息工程学院）
Imperial College London（帝国理工学院）；New York University（纽约大学）；Twitter（推特）
Intrinsic graph convolution operators with differentiable kernel functions play a crucial role in analyzing 3D shape meshes. In this paper, we present a fast and efficient intrinsic mesh convolution operator that does not rely on the intricate design of kernel function. We explicitly formulate the order of aggregating neighboring vertices, instead of learning weights between nodes, and then a fully connected layer follows to fuse local geometric structure information with vertex features. We provide extensive evidence showing that models based on this convolution operator are easier to train, …
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Shenzhen University（深圳大学）；Renmin University of China（中国人民大学）
Fusing medical images and the corresponding 3D shape representation can provide complementary information and microstructure details to improve the operational performance and accuracy in brain surgery. However, compared to the substantial image data, it is almost impossible to obtain the intraoperative 3D shape information by using physical methods such as sensor scanning, especially in minimally invasive surgery and robot-guided surgery. In this paper, a general generative adversarial network (GAN) architecture based on graph convolutional networks is proposed to reconstruct the 3D point clo…
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；IIT Bombay（印度理工学院孟买分校）；Adobe Research（Adobe 研究院）
We present a new local descriptor for 3D shapes, directly applicable to a wide range of shape analysis problems such as point correspondences, semantic segmentation, affordance prediction, and shape-to-scan matching. The descriptor is produced by a convolutional network that is trained to embed geometrically and semantically similar points close to one another in descriptor space. The network processes surface neighborhoods around points on a shape that are captured at multiple scales by a succession of progressively zoomed out views, taken from carefully selected camera positions. We leverage…
