Statistical Hand Shape Modeling from Clinical CT Scans Using Deep Learning and Implicit Skinning
Dept. of Computer Science and Engineering（计算机科学与工程系）；Özyeğin University（奥兹耶尼大学）；Dept. of Artificial Intelligence and Data Engineering（人工智能与数据工程系）；Osteoid Inc.（Osteoid公司）
Gokce Guven, Hasan Fehmi Ates, Deniz Karasahin, Kaan Erdogan
准确的分割和手部解剖的统计形状建模对医学诊断、人机工程学和生物力学有重要影响。本研究提出了一种AI辅助的重建流程，用于从1,271例肘至手（e2h-CT）计算机断层扫描中分割和分析手部解剖结构。首先使用基于Pix2Pix的条件生成对抗网络去除CT体积中的石膏和背景伪影。清洁后的扫描随后在3D Slicer中处理，提取皮肤和骨掩膜，并将其转换为封闭曲面网格模型。分割的骨网格用于构建骨骼表示，使隐式皮肤能够将所有手模型对齐到标准化的解剖配置。随后，使用Geodesic Based Coherent Point Drift++（GBCPD++）算法对手部皮肤表面进行非刚性配准，以在不同受试者之间建立点对应关系。然后对配准后的模型应用主成分分析（PCA）以量化解剖形状的变异性。Pix2Pix预处理阶段在保留测试集上实现了Dice系数为0.9856和IoU为0.9720。统计建模在90例扫描中进行，其中手指完全可见且解剖上分离。所得的统计形状分布与美国陆军人体测量调查（ANSUR II）有很强的一致性，支持重建模型的解剖有效性。所提出的方法在生物力学建模、人机工程学优化、假肢设计和精准医疗诊断方面具有显著潜力。
英文摘要：
Accurate segmentation and statistical shape modeling of hand anatomy have significant implications for medical diagnostics, ergonomics, and biomechanics. This study proposes an AI-assisted reconstruction pipeline for segmenting and analyzing hand anatomy from 1,271 elbow-to-hand (e2h-CT) computed tomography scans. A Pix2Pix-based conditional generative adversarial network is first employed to remove plaster cast and background artifacts from CT volumes. The cleaned scans are then processed in 3D Slicer to extract skin and bone masks, which are converted into closed-surface mesh models. Segmented bone meshes are used to construct skeletal representations, enabling implicit skinning to align all hand models into a standardized anatomical configuration. Subsequently, non-rigid registration is performed on the hand skin surfaces using the Geodesic Based Coherent Point Drift++ (GBCPD++) algorithm to establish point-wise correspondence across subjects. Principal Component Analysis (PCA) is then applied to the registered models to quantify anatomical shape variability. The Pix2Pix preprocessing stage achieved a Dice coefficient of 0.9856 and an IoU of 0.9720 on the held-out test set. Statistical modeling was performed on a subset of 90 scans in which the fingers were fully visible and anatomically separated. The resulting statistical shape distributions demonstrate strong agreement with the U.S. Army Anthropometric Survey (ANSUR II), supporting the anatomical validity of the reconstructed models. The proposed methodology demonstrates significant potential for advancing biomechanical modeling, ergonomic optimization, prosthetic design, and precision medical diagnostics.
ETH Zürich（苏黎世联邦理工学院）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Tübingen（图宾根大学）
Neural implicit surface representations have emerged as a promising paradigm to capture 3D shapes in a continuous and resolution-independent manner. However, adapting them to articulated shapes is non-trivial. Existing approaches learn a backward warp field that maps deformed to canonical points. However, this is problematic since the backward warp field is pose dependent and thus requires large amounts of data to learn. To address this, we introduce SNARF, which combines the advantages of linear blend skinning (LBS) for polygonal meshes with those of neural implicit surfaces by learning a for…
University of Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）
We present a novel solution to the garment animation problem through deep learning. Our contribution allows animating any template outfit with arbitrary topology and geometric complexity. Recent works develop models for garment edition, resizing and animation at the same time by leveraging the support body model (encoding garments as body homotopies). This leads to complex engineering solutions that suffer from scalability, applicability and compatibility. By limiting our scope to garment animation only, we are able to propose a simple model that can animate any outfit, independently of its to…
Uber Advanced Technologies Group（优步先进技术集团）；University of Toronto（多伦多大学）；University of Southern California（南加州大学）；Massachusetts Institute of Techonology（麻省理工学院）
Constructing and animating humans is an important component for building virtual worlds in a wide variety of applications such as virtual reality or robotics testing in simulation. As there are exponentially many variations of humans with different shape, pose and clothing, it is critical to develop methods that can automatically reconstruct and animate humans at scale from real world data. Towards this goal, we represent the pedestrian's shape, pose and skinning weights as neural implicit functions that are directly learned from data. This representation enables us to handle a wide variety of…
MoE-dqINR: A Unified Mixture-of-Experts Implicit Neural Representation Framework for Scan-Specific Dynamic and Quantitative MRI Reconstruction
Department of Bioengineering and I-X, Imperial College London（生物工程系和I-X，帝国理工学院伦敦分校）；Cardiovascular Research Centre, Royal Brompton Hospital（心脏血管研究中心，皇家布隆特医院）；National Heart and Lung Institute, Imperial College London（国家心脏和肺研究所，帝国理工学院伦敦分校）；School of Biomedical Engineering & Imaging Sciences, King’s College London（生物医学工程与成像科学学院，伦敦国王学院）；Shanghai Pudong Hospital and Human Phenome Institute, Fudan University（上海浦东医院和人类表型研究所，复旦大学）；International Human Phenome Institute (Shanghai), Shanghai, China（国际人类表型研究所（上海），上海，中国）
We present FITE, a First-Implicit-Then-Explicit framework for modeling human avatars in clothing. Our framework first learns implicit surface templates representing the coarse clothing topology, and then employs the templates to guide the generation of point sets which further capture pose-dependent clothing deformations such as wrinkles. Our pipeline incorporates the merits of both implicit and explicit representations, namely, the ability to handle varying topology and the ability to efficiently capture fine details. We also propose diffused skinning to facilitate template training especiall…
In this paper, we introduce a novel approach to learn a 3D face model using a joint-based face rig and a neural skinning network. Thanks to the joint-based representation, our model enjoys some significant advantages over prior blendshape-based models. First, it is very compact such that we are orders of magnitude smaller while still keeping strong modeling capacity. Second, because each joint has its semantic meaning, interactive facial geometry editing is made easier and more intuitive. Third, through skinning, our model supports adding mouth interior and eyes, as well as accessories (hair, …
Accelerated Patient-specific Non-Cartesian MRI Reconstruction using Implicit Neural Representations
University of California, San Francisco（加利福尼亚大学旧金山分校）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Keck School of Medicine, University of Southern California（南加州大学凯克医学院）；University of San Francisco（旧金山大学）
Institut de Robòtica i Informàtica Industrial, CSIC-UPC（西班牙国家研究委员会-加泰罗尼亚理工大学工业机器人与信息技术研究所）；Reality Labs, Meta（Meta现实实验室）
This paper proposes a do-it-all neural model of human hands, named LISA. The model can capture accurate hand shape and appearance, generalize to arbitrary hand subjects, provide dense surface correspondences, be reconstructed from images in the wild and easily animated. We train LISA by minimizing the shape and appearance losses on a large set of multi-view RGB image sequences annotated with coarse 3D poses of the hand skeleton. For a 3D point in the hand local coordinate, our model predicts the color and the signed distance with respect to each hand bone independently, and then combines the p…
ETH Zürich（苏黎世联邦理工学院）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Parametric 3D body models like SMPL only represent minimally-clothed people and are hard to extend to clothing because they have a fixed mesh topology and resolution. To address these limitations, recent work uses implicit surfaces or point clouds to model clothed bodies. While not limited by topology, such methods still struggle to model clothing that deviates significantly from the body, such as skirts and dresses. This is because they rely on the body to canonicalize the clothed surface by reposing it to a reference shape. Unfortunately, this process is poorly defined when clothing is far f…
RibPull: Implicit Occupancy Fields and Medial Axis Extraction for CT Ribcage Scans
Electrical and Computer Engineering Department, University of California, Santa Cruz（加州大学圣克ruz分校电子与计算机工程系）；Inria, Univ. Rennes, CNRS, IRISA, M2S, Rennes（Inria，里尔大学，CNRS，IRISA，M2S，里尔）；LIRIS - CNRS - Université Claude Bernard Lyon 1（LIRIS-CNRS-里尔1大学）；Computer Science Engineering Department, University of California, Santa Cruz（加州大学圣克ruz分校计算机科学工程系）
SGAM: Shared Gaussian Geometry with Implicit Amplitude Modeling for Scan-Specific 3D Multi-Contrast MRI Reconstruction
Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院劳特伯生物医学成像研究中心）
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Saarland University（萨尔大学）；Google Research（谷歌研究院）
We address the problem of fitting 3D human models to 3D scans of dressed humans. Classical methods optimize both the data-to-model correspondences and the human model parameters (pose and shape), but are reliable only when initialized close to the solution. Some methods initialize the optimization based on fully supervised correspondence predictors, which is not differentiable end-to-end, and can only process a single scan at a time. Our main contribution is LoopReg, an end-to-end learning framework to register a corpus of scans to a common 3D human model. The key idea is to create a self-supe…
Parallel imaging is a widely-used technique to accelerate magnetic resonance imaging (MRI). However, current methods still perform poorly in reconstructing artifact-free MRI images from highly undersampled k-space data. Recently, implicit neural representation (INR) has emerged as a new deep learning paradigm for learning the internal continuity of an object. In this study, we adopted INR to parallel MRI reconstruction. The MRI image was modeled as a continuous function of spatial coordinates. This function was parameterized by a neural network and learned directly from the measured k-space it…
We propose a probabilistic shape completion method extended to the continuous geometry of large-scale 3D scenes. Real-world scans of 3D scenes suffer from a considerable amount of missing data cluttered with unsegmented objects. The problem of shape completion is inherently ill-posed, and high-quality result requires scalable solutions that consider multiple possible outcomes. We employ the Generative Cellular Automata that learns the multi-modal distribution and transform the formulation to process large-scale continuous geometry. The local continuous shape is incrementally generated as a spa…
High-Fidelity 3D Tooth Reconstruction by Fusing Intraoral Scans and CBCT Data via a Deep Implicit Representation
INSA-Lyon, UCBL, CNRS, Inserm, CREATIS UMR 5220, U1294, F-69621, Lyon, France（里昂INSA大学、UCBL、CNRS、Inserm、CREATIS UMR 5220、U1294、法国里昂）；National Institute of Informatics (NII), Tokyo, Japan（日本东京国立信息研究所）；Hospices Civils de Lyon, PAM Odontologie, Lyon, France（里昂医院、PAM牙科）
Recently, several works have addressed modeling of 3D shapes using deep neural networks to learn implicit surface representations. Up to now, the majority of works have concentrated on reconstruction quality, paying little or no attention to model size or training time. This work proposes LightSAL, a novel deep convolutional architecture for learning 3D shapes; the proposed work concentrates on efficiency both in network training time and resulting model size. We build on the recent concept of Sign Agnostic Learning for training the proposed network, relying on signed distance fields, with uns…
University of Bristol（布里斯托大学）；Zhejiang University（浙江大学）
We propose a novel end-to-end RGB-D SLAM, iDF-SLAM, which adopts a feature-based deep neural tracker as the front-end and a NeRF-style neural implicit mapper as the back-end. The neural implicit mapper is trained on-the-fly, while though the neural tracker is pretrained on the ScanNet dataset, it is also finetuned along with the training of the neural implicit mapper. Under such a design, our iDF-SLAM is capable of learning to use scene-specific features for camera tracking, thus enabling lifelong learning of the SLAM system. Both the training for the tracker and the mapper are self-supervised…
Establishing a correspondence between two non-rigidly deforming shapes is one of the most fundamental problems in visual computing. Existing methods often show weak resilience when presented with challenges innate to real-world data such as noise, outliers, self-occlusion etc. On the other hand, auto-decoders have demonstrated strong expressive power in learning geometrically meaningful latent embeddings. However, their use in \emph{shape analysis} has been limited. In this paper, we introduce an approach based on an auto-decoder framework, that learns a continuous shape-wise deformation field…
South China University of Technology（华南理工大学）；Pazhou Lab（琶洲实验室）；Peng Cheng Lab（鹏城实验室）；Southern University of Science and Technology（南方科技大学）
Shape modeling and reconstruction from raw point clouds of objects stand as a fundamental challenge in vision and graphics research. Classical methods consider analytic shape priors; however, their performance degraded when the scanned points deviate from the ideal conditions of cleanness and completeness. Important progress has been recently made by data-driven approaches, which learn global and/or local models of implicit surface representations from auxiliary sets of training shapes. Motivated from a universal phenomenon that self-similar shape patterns of local surface patches repeat acros…
École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；Neural Concept；Interl Labs（Interl实验室）；Meta Reality Labs（Meta现实实验室）
Geometric Deep Learning has recently made striking progress with the advent of continuous deep implicit fields. They allow for detailed modeling of watertight surfaces of arbitrary topology while not relying on a 3D Euclidean grid, resulting in a learnable parameterization that is unlimited in resolution. Unfortunately, these methods are often unsuitable for applications that require an explicit mesh-based surface representation because converting an implicit field to such a representation relies on the Marching Cubes algorithm, which cannot be differentiated with respect to the underlying imp…
3D textured shape recovery from partial scans is crucial for many real-world applications. Existing approaches have demonstrated the efficacy of implicit function representation, but they suffer from partial inputs with severe occlusions and varying object types, which greatly hinders their application value in the real world. This technical report presents our approach to address these limitations by incorporating learned geometric priors. To this end, we generate a SMPL model from learned pose prediction and fuse it into the partial input to add prior knowledge of human bodies. We also propo…
Reconstructing 3D human body shapes from 3D partial textured scans remains a fundamental task for many computer vision and graphics applications -- e.g., body animation, and virtual dressing. We propose a new neural network architecture for 3D body shape and high-resolution texture completion -- BCom-Net -- that can reconstruct the full geometry from mid-level to high-level partial input scans. We decompose the overall reconstruction task into two stages - first, a joint implicit learning network (SCom-Net and TCom-Net) that takes a voxelized scan and its occupancy grid as input to reconstruct…
University of Oxford（牛津大学）；Leiden University Medical Center（莱顿大学医学中心）
The objective of this work is to achieve sensorless reconstruction of a 3D volume from a set of 2D freehand ultrasound images with deep implicit representation. In contrast to the conventional way that represents a 3D volume as a discrete voxel grid, we do so by parameterizing it as the zero level-set of a continuous function, i.e. implicitly representing the 3D volume as a mapping from the spatial coordinates to the corresponding intensity values. Our proposed model, termed as ImplicitVol, takes a set of 2D scans and their estimated locations in 3D as input, jointly refining the estimated 3D …
Mass General Brigham, Harvard Medical School, MA, United States（马萨诸塞州总医院、哈佛医学院）；Department of Statistics, University of South Carolina, SC, United States（统计学系，南卡罗来纳大学）；Department of Anatomy and Neurobiology, Boston University Chobanian & Avedisian School of Medicine, MA, United States（解剖学与神经生物学系，波士顿大学钱德林与阿维迪亚安医学学院）；Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, MA, United States（阿提诺拉A·马丁诺斯生物医学成像中心，马萨诸塞总医院）；Department of Radiology, Harvard Medical School, MA, United States（放射学系，哈佛医学院）；Harvard/MIT Health Sciences and Technology, MA, United States（哈佛/麻省理工健康科学与技术）
ImplicitCell: Resolution Cell Modeling of Joint Implicit Volume Reconstruction and Pose Refinement in Freehand 3D Ultrasound
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Tongren Hospital, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院附属同仁医院）
Single-Subject Multi-View MRI Super-Resolution via Implicit Neural Representations
Department of Radiology, Weill Cornell Medicine, New York, USA（韦尔·科恩医学系放射科，纽约，美国）；School of Electrical and Computer Engineering, Cornell University（康奈尔大学电气与计算机工程学院）；Cornell Tech, New York, USA（康奈尔科技，纽约，美国）；Weill Cornell Graduate School of Medical Sciences, New York, USA（韦尔·科恩医学科学研究生院，纽约，美国）
Simon Fraser University（西蒙弗雷泽大学）；Reality Labs Research（现实实验室研究院）
Neural fields such as implicit surfaces have recently enabled avatar modeling from raw scans without explicit temporal correspondences. In this work, we exploit autoregressive modeling to further extend this notion to capture dynamic effects, such as soft-tissue deformations. Although autoregressive models are naturally capable of handling dynamics, it is non-trivial to apply them to implicit representations, as explicit state decoding is infeasible due to prohibitive memory requirements. In this work, for the first time, we enable autoregressive modeling of implicit avatars. To reduce the mem…
University of Toronto（多伦多大学）；McGill University（麦吉尔大学）
A neural implicit outputs a number indicating whether the given query point in space is inside, outside, or on a surface. Many prior works have focused on _latent-encoded_ neural implicits, where a latent vector encoding of a specific shape is also fed as input. While affording latent-space interpolation, this comes at the cost of reconstruction accuracy for any _single_ shape. Training a specific network for each 3D shape, a _weight-encoded_ neural implicit may forgo the latent vector and focus reconstruction accuracy on the details of a single shape. While previously considered as an interme…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ETH Zürich（苏黎世联邦理工学院）
Currently it requires an artist to create 3D human avatars with realistic clothing that can move naturally. Despite progress on 3D scanning and modeling of human bodies, there is still no technology that can easily turn a static scan into an animatable avatar. Automating the creation of such avatars would enable many applications in games, social networking, animation, and AR/VR to name a few. The key problem is one of representation. Standard 3D meshes are widely used in modeling the minimally-clothed body but do not readily capture the complex topology of clothing. Recent interest has shifte…
Magnetic resonance (MR) images collected in 2D clinical protocols typically have large inter-slice spacing, resulting in high in-plane resolution and reduced through-plane resolution. Super-resolution technique can enhance the through-plane resolution of MR images to facilitate downstream visualization and computer-aided diagnosis. However, most existing works train the super-resolution network at a fixed scaling factor, which is not friendly to clinical scenes of varying inter-slice spacing in MR scanning. Inspired by the recent progress in implicit neural representation, we propose a Spatial…
Delft University of Technology（代尔夫特理工大学）；Technical University of Munich（慕尼黑工业大学）
While three-dimensional (3D) building models play an increasingly pivotal role in many real-world applications, obtaining a compact representation of buildings remains an open problem. In this paper, we present a novel framework for reconstructing compact, watertight, polygonal building models from point clouds. Our framework comprises three components: (a) a cell complex is generated via adaptive space partitioning that provides a polyhedral embedding as the candidate set; (b) an implicit field is learned by a deep neural network that facilitates building occupancy estimation; (c) a Markov ra…
Recently, neural networks have been used as implicit representations for surface reconstruction, modelling, learning, and generation. So far, training neural networks to be implicit representations of surfaces required training data sampled from a ground-truth signed implicit functions such as signed distance or occupancy functions, which are notoriously hard to compute. In this paper we introduce Sign Agnostic Learning (SAL), a deep learning approach for learning implicit shape representations directly from raw, unsigned geometric data, such as point clouds and triangle soups. We have tested …
TU Delft（代尔夫特理工大学）；Mercedes-Benz AG（梅赛德斯-奔驰集团）；Esslingen University of Applied Sciences（埃斯林根应用科学大学）
Semantic scene completion is the task of jointly estimating 3D geometry and semantics of objects and surfaces within a given extent. This is a particularly challenging task on real-world data that is sparse and occluded. We propose a scene segmentation network based on local Deep Implicit Functions as a novel learning-based method for scene completion. Unlike previous work on scene completion, our method produces a continuous scene representation that is not based on voxelization. We encode raw point clouds into a latent space locally and at multiple spatial resolutions. A global scene complet…
We present the first deep implicit 3D morphable model (i3DMM) of full heads. Unlike earlier morphable face models it not only captures identity-specific geometry, texture, and expressions of the frontal face, but also models the entire head, including hair. We collect a new dataset consisting of 64 people with different expressions and hairstyles to train i3DMM. Our approach has the following favorable properties: (i) It is the first full head morphable model that includes hair. (ii) In contrast to mesh-based models it can be trained on merely rigidly aligned scans, without requiring difficult…
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
In this work we target a learnable output representation that allows continuous, high resolution outputs of arbitrary shape. Recent works represent 3D surfaces implicitly with a Neural Network, thereby breaking previous barriers in resolution, and ability to represent diverse topologies. However, neural implicit representations are limited to closed surfaces, which divide the space into inside and outside. Many real world objects such as walls of a scene scanned by a sensor, clothing, or a car with inner structures are not closed. This constitutes a significant barrier, in terms of data pre-pr…
