DeforHMR 提出一种基于回归的单目 3D 人体网格恢复框架，通过在 transformer 解码器中引入与查询无关的可变形交叉注意力机制，从冻结的 ViT 编码器提取视觉特征进行回归，在 3DPW 和 RICH 基准上达到 SOTA 性能。
摘要：
Human Mesh Recovery (HMR) 是一个重要且具挑战性的问题，其应用涵盖动作捕捉、增强现实和生物力学等多个领域。从单张图像中准确预测人体姿态参数仍然是一项极具挑战性的 3D 计算机视觉任务。在本研究中，我们提出了 DeforHMR，这是一种新颖的基于回归的单目 HMR 框架，旨在利用可变形注意力 transformer 来增强人体姿态参数的预测。DeforHMR 在 transformer 解码器内部利用一种新颖的与查询无关的可变形交叉注意力机制，有效地对从冻结的预训练 vision transformer (ViT) 编码器中提取的视觉特征进行回归。所提出的可变形交叉注意力机制使得模型能够以数据驱动的方式更灵活地关注相关的空间特征。借助具备空间细粒度注意力的 transformer 解码器，DeforHMR 在广泛使用的 3D HMR 基准 3DPW 和 RICH 上，在基于单帧回归的方法中取得了 state-of-the-art 的性能。通过可变形注意力推动 3D 人体网格恢复领域的边界，我们为计算机视觉中从大型预训练视觉编码器解码局部空间信息引入了一种新颖且有效的范式。
英文摘要：
Human Mesh Recovery (HMR) is an important yet challenging problem with applications across various domains including motion capture, augmented reality, and biomechanics. Accurately predicting human pose parameters from a single image remains a challenging 3D computer vision task. In this work, we introduce DeforHMR, a novel regression-based monocular HMR framework designed to enhance the prediction of human pose parameters using deformable attention transformers. DeforHMR leverages a novel query-agnostic deformable cross-attention mechanism within the transformer decoder to effectively regress the visual features extracted from a frozen pretrained vision transformer (ViT) encoder. The proposed deformable cross-attention mechanism allows the model to attend to relevant spatial features more flexibly and in a data-dependent manner. Equipped with a transformer decoder capable of spatially-nuanced attention, DeforHMR achieves state-of-the-art performance for single-frame regression-based methods on the widely used 3D HMR benchmarks 3DPW and RICH. By pushing the boundary on the field of 3D human mesh recovery through deformable attention, we introduce an new, effective paradigm for decoding local spatial information from large pretrained vision encoders in computer vision.
This paper presents a novel approach to reconstruct complete 3D deformable models over time by a single depth camera. These are the steps employed for deforming objects from single depth camera. The partial surfaces reconstructed from various times of capture are assembled together to form a complete 3D surface. A mesh warping algorithm is used to align different partial surfaces based on linear mesh deformation. A volumetric method is then applied to combine partial surfaces, fix missing holes and smooth alignment errors.
BridgeSplat: Bidirectionally Coupled CT and Non-Rigid Gaussian Splatting for Deformable Intraoperative Surgical Navigation
Computer Aided Medical Procedures, TU Munich, Munich, Germany（慕尼黑技术大学计算机辅助医疗程序部）；Department of General, Visceral, and Transplantation Surgery, Hospital of the LMU Munich, Ludwig-Maximilians-Universität (LMU), Munich, Germany（慕尼黑路德维希-马克西米利安大学（LMU）普通外科、 visceral外科及移植外科部门）；Harvard Medical School, Brigham and Women's Hospital, Boston, MA, USA（哈佛医学院布里特妇女医院）
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；City University of Hong Kong（香港城市大学）；Cardiff University（卡迪夫大学）；Simon Fraser University（西蒙弗雷泽大学）
We introduce SDM-NET, a deep generative neural network which produces structured deformable meshes. Specifically, the network is trained to generate a spatial arrangement of closed, deformable mesh parts, which respect the global part structure of a shape collection, e.g., chairs, airplanes, etc. Our key observation is that while the overall structure of a 3D shape can be complex, the shape can usually be decomposed into a set of parts, each homeomorphic to a box, and the finer-scale geometry of the part can be recovered by deforming the box. The architecture of SDM-NET is that of a two-level …
Department of Computer Science and Electrical & Computer Engineering, University of Maryland at College Park（马里兰大学学院公园分校计算机科学与电子与计算机工程系）；Department of Computer Science, University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校计算机科学系）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
We address the problem of accelerating thin-shell deformable object simulations by dimension reduction. We present a new algorithm to embed a high-dimensional configuration space of deformable objects in a low-dimensional feature space, where the configurations of objects and feature points have approximate one-to-one mapping. Our key technique is a graph-based convolutional neural network (CNN) defined on meshes with arbitrary topologies and a new mesh embedding approach based on physics-inspired loss term. We have applied our approach to accelerate high-resolution thin shell simulations corr…
Max Planck Institute for Informatics, Saarland Informatics Campus（马克斯·普朗克信息学研究所，萨尔兰信息学园区）；Stanford University（斯坦福大学）
Mesh autoencoders are commonly used for dimensionality reduction, sampling and mesh modeling. We propose a general-purpose DEep MEsh Autoencoder (DEMEA) which adds a novel embedded deformation layer to a graph-convolutional mesh autoencoder. The embedded deformation layer (EDL) is a differentiable deformable geometric proxy which explicitly models point displacements of non-rigid deformations in a lower dimensional space and serves as a local rigidity regularizer. DEMEA decouples the parameterization of the deformation from the final mesh resolution since the deformation is defined over a lowe…
Imperial College London（帝国理工学院）；Università della Svizzera italiana（瑞士意大利语区大学）；Ariel AI（Ariel人工智能公司）；Twitter, Inc.（推特公司）
Monocular 3D reconstruction of deformable objects, such as human body parts, has been typically approached by predicting parameters of heavyweight linear models. In this paper, we demonstrate an alternative solution that is based on the idea of encoding images into a latent non-linear representation of meshes. The prior on 3D hand shapes is learned by training an autoencoder with intrinsic graph convolutions performed in the spectral domain. The pre-trained decoder acts as a non-linear statistical deformable model. The latent parameters that reconstruct the shape and articulated pose of hands …
We present a learning framework for recovering the 3D shape, camera, and texture of an object from a single image. The shape is represented as a deformable 3D mesh model of an object category where a shape is parameterized by a learned mean shape and per-instance predicted deformation. Our approach allows leveraging an annotated image collection for training, where the deformable model and the 3D prediction mechanism are learned without relying on ground-truth 3D or multi-view supervision. Our representation enables us to go beyond existing 3D prediction approaches by incorporating texture inf…
Intensity modulated radiotherapy (IMRT) is one of the most common modalities for treating cancer patients. One of the biggest challenges is precise treatment delivery that accounts for varying motion patterns originating from free-breathing. Currently, image-guided solutions for IMRT is limited to 2D guidance due to the complexity of 3D tracking solutions. We propose a novel end-to-end attention graph neural network model that generates in real-time a triangular shape of the liver based on a reference segmentation obtained at the preoperative phase and a 2D MRI coronal slice taken during the t…
Leiden University Medical Center（莱顿大学医学中心）；Centrum Wiskunde & Informatica（荷兰数学与计算机科学研究院）
Reliably and physically accurately transferring information between images through deformable image registration with large anatomical differences is an open challenge in medical image analysis. Most existing methods have two key shortcomings: first, they require extensive up-front parameter tuning to each specific registration problem, and second, they have difficulty capturing large deformations and content mismatches between images. There have however been developments that have laid the foundation for potential solutions to both shortcomings. Towards the first shortcoming, a multi-objectiv…
Graduate School of Informatics, Kyoto University（京都大学信息学研究科）；Graduate School of Medicine, Kyoto University（京都大学医学研究科）
Organ shape reconstruction based on a single-projection image during treatment has wide clinical scope, e.g., in image-guided radiotherapy and surgical guidance. We propose an image-to-graph convolutional network that achieves deformable registration of a 3D organ mesh for a single-viewpoint 2D projection image. This framework enables simultaneous training of two types of transformation: from the 2D projection image to a displacement map, and from the sampled per-vertex feature to a 3D displacement that satisfies the geometrical constraint of the mesh structure. Assuming application to radiati…
Saarland University（萨尔大学）；MPI for Informatics（马克斯·普朗克信息学研究所）
3D human motion capture from monocular RGB images respecting interactions of a subject with complex and possibly deformable environments is a very challenging, ill-posed and under-explored problem. Existing methods address it only weakly and do not model possible surface deformations often occurring when humans interact with scene surfaces. In contrast, this paper proposes MoCapDeform, i.e., a new framework for monocular 3D human motion capture that is the first to explicitly model non-rigid deformations of a 3D scene for improved 3D human pose estimation and deformable environment reconstruct…
The last several years have seen significant progress in using depth cameras for tracking articulated objects such as human bodies, hands, and robotic manipulators. Most approaches focus on tracking skeletal parameters of a fixed shape model, which makes them insufficient for applications that require accurate estimates of deformable object surfaces. To overcome this limitation, we present a 3D model-based tracking system for articulated deformable objects. Our system is able to track human body pose and high resolution surface contours in real time using a commodity depth sensor and GPU hardw…
Georgia Institute of Technology（佐治亚理工学院）；Facebook Reality Labs Research（Facebook现实实验室研究）；Intel Labs（英特尔实验室）
Physical contact between hands and objects plays a critical role in human grasps. We show that optimizing the pose of a hand to achieve expected contact with an object can improve hand poses inferred via image-based methods. Given a hand mesh and an object mesh, a deep model trained on ground truth contact data infers desirable contact across the surfaces of the meshes. Then, ContactOpt efficiently optimizes the pose of the hand to achieve desirable contact using a differentiable contact model. Notably, our contact model encourages mesh interpenetration to approximate deformable soft tissue in…
In this work, we focus on the task of learning and representing dense correspondences in deformable object categories. While this problem has been considered before, solutions so far have been rather ad-hoc for specific object types (i.e., humans), often with significant manual work involved. However, scaling the geometry understanding to all objects in nature requires more automated approaches that can also express correspondences between related, but geometrically different objects. To this end, we propose a new, learnable image-based representation of dense correspondences. Our model predic…
International Digital Economy Academy (IDEA)（国际数字经济研究院（IDEA））；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）
Whole-body mesh recovery aims to estimate the 3D human body, face, and hands parameters from a single image. It is challenging to perform this task with a single network due to resolution issues, i.e., the face and hands are usually located in extremely small regions. Existing works usually detect hands and faces, enlarge their resolution to feed in a specific network to predict the parameter, and finally fuse the results. While this copy-paste pipeline can capture the fine-grained details of the face and hands, the connections between different parts cannot be easily recovered in late fusion,…
UC Berkeley（加州大学伯克利分校）；Mitsubishi Electric Research Laboratories (MERL)（三菱电机研究实验室（MERL））
Vision-based tactile sensors typically utilize a deformable elastomer and a camera mounted above to provide high-resolution image observations of contacts. Obtaining accurate volumetric meshes for the deformed elastomer can provide direct contact information and benefit robotic grasping and manipulation. This paper focuses on learning to synthesize the volumetric mesh of the elastomer based on the image imprints acquired from vision-based tactile sensors. Synthetic image-mesh pairs and real-world images are gathered from 3D finite element methods (FEM) and physical sensors, respectively. A gra…
End-to-End 4D Heart Mesh Recovery Across Full-Stack and Sparse Cardiac MRI
School of Computer and Communication Science, EPFL（计算机与通信科学学院，瑞士联邦理工学院）；ELLIS Institute Finland, Aalto University, Finland（芬兰ELLIS研究所，阿尔托大学，芬兰）；University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）；Center for Interventional MRI, CHUV（介入MRI中心，CHUV）
DELIFFAS: Deformable Light Fields for Fast Avatar Synthesis
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of Pennsylvania（宾夕法尼亚大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Saarbrücken Research Center for Visual Computing, Interaction and AI（萨尔布吕肯视觉计算、交互与人工智能研究中心）
