Geometry-Conditioned Diffusion for Occlusion-Robust In-Bed Pose Estimation
Department of Computer Science, University of Verona（威尼斯大学计算机科学系）；EVS - Embedded Vision Systems Srl（嵌入式视觉系统股份有限公司）
Navid Aslankhani Khameneh, Marco Carletti, Cigdem Beyan
备注：
This is the preprint version of the paper. The final version has been accepted for publication in the Proceedings of the 20th IEEE International Conference on Automatic Face and Gesture Recognition (IEEE FG 2026)
Robust in-bed human pose estimation under blanket occlusion remains challenging due to the scarcity of reliable labeled training data for heavily covered poses. Existing approaches rely on multi-modal sensing or image-to-image translation frameworks that remain conditioned on visible source imagery, limiting scalability and pose diversity. In this work, we reformulate occlusion-aware augmentation as a geometry-conditioned generative modeling task. We conduct a systematic comparison of deterministic masking, unpaired translation, paired diffusion-based translation, and a proposed pose-conditioned Latent Diffusion Model (Pose-LDM). Unlike image-guided methods, Pose-LDM synthesizes blanket-covered images directly from skeletal keypoints, eliminating dependence on paired supervision and pixel-level source-image conditioning while enabling generation from arbitrary pose inputs. All augmentation strategies are evaluated through their impact on downstream pose estimation under a fixed backbone. Pose- LDM achieves the highest strict localization accuracy under severe occlusion while maintaining overall detection performance comparable to paired diffusion models, approaching the performance of fully supervised training. These results demonstrate that geometry-conditioned diffusion provides an effective and supervision-efficient pathway toward occlusion-robust inbed pose estimation without modifying the sensing pipeline. The code is available at: github.com/navidTerraNova/ GeoDiffPose.
Georgia Institute of Technology（佐治亚理工学院）；Stanford University（斯坦福大学）
People spend a substantial part of their lives at rest in bed. 3D human pose and shape estimation for this activity would have numerous beneficial applications, yet line-of-sight perception is complicated by occlusion from bedding. Pressure sensing mats are a promising alternative, but training data is challenging to collect at scale. We describe a physics-based method that simulates human bodies at rest in a bed with a pressure sensing mat, and present PressurePose, a synthetic dataset with 206K pressure images with 3D human poses and shapes. We also present PressureNet, a deep learning model…
ST Engineering - NTU Corporate Lab（ST工程-南洋理工大学企业实验室）；Nanyang Technological University（南洋理工大学）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）；City University of Hong Kong（香港城市大学）
The light field faithfully records the spatial and angular configurations of the scene, which facilitates a wide range of imaging possibilities. In this work, we propose an LF synthesis algorithm which renders high quality novel LF views far outside the range of angular baselines of the given references. A stratified synthesis strategy is adopted which parses the scene content based on stratified disparity layers and across a varying range of spatial granularities. Such a stratified methodology proves to help preserve scene structures over large perspective shifts, and it provides informative …
Johns Hopkins University（约翰斯·霍普金斯大学）；Florida International University（佛罗里达国际大学）
Human behavioral monitoring during sleep is essential for various medical applications. Majority of the contactless human pose estimation algorithms are based on RGB modality, causing ineffectiveness in in-bed pose estimation due to occlusions by blankets and varying illumination conditions. Long-wavelength infrared (LWIR) modality based pose estimation algorithms overcome the aforementioned challenges; however, ground truth pose generations by a human annotator under such conditions are not feasible. A feasible solution to address this issue is to transfer the knowledge learned from images wi…
Contact pressure between the human body and its surroundings has important implications. For example, it plays a role in comfort, safety, posture, and health. We present a method that infers contact pressure between a human body and a mattress from a depth image. Specifically, we focus on using a depth image from a downward facing camera to infer pressure on a body at rest in bed occluded by bedding, which is directly applicable to the prevention of pressure injuries in healthcare. Our approach involves augmenting a real dataset with synthetic data generated via a soft-body physics simulation …
Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；University of Chinese Academy of Sciences (UCAS)（中国科学院大学）
In recent years, virtual makeup applications have become more and more popular. However, it is still challenging to propose a robust makeup transfer method in the real-world environment. Current makeup transfer methods mostly work well on good-conditioned clean makeup images, but transferring makeup that exhibits shadow and occlusion is not satisfying. To alleviate it, we propose a novel makeup transfer method, called 3D-Aware Shadow and Occlusion Robust GAN (SOGAN). Given the source and the reference faces, we first fit a 3D face model and then disentangle the faces into shape and texture. In…
University of Massachusetts, Amherst（马萨诸塞大学阿默斯特分校）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Google Research（谷歌研究院）；University of Amsterdam（阿姆斯特丹大学）；Microsoft Research（微软研究院）
Humans spend vast hours in bed -- about one-third of the lifetime on average. Besides, a human at rest is vital in many healthcare applications. Typically, humans are covered by a blanket when resting, for which we propose a multimodal approach to uncover the subjects so their bodies at rest can be viewed without the occlusion of the blankets above. We propose a pyramid scheme to effectively fuse the different modalities in a way that best leverages the knowledge captured by the multimodal sensors. Specifically, the two most informative modalities (i.e., depth and infrared images) are first fu…
CloseUpShot: Close-up Novel View Synthesis from Sparse-views via Point-conditioned Diffusion Model
Shenzhen Future Network of Intelligence Institute (FNii-Shenzhen)（深圳未来网络智能研究院（FNii-深圳））；Chinese University of Hong Kong at Shenzhen (CUHKSZ)（香港中文大学（深圳））；Sun Yat-sen University（中山大学）
NEC Laboratories America（NEC美国实验室）；Michigan State University（密歇根州立大学）；University of California, San Diego（加利福尼亚大学圣迭戈分校）
Despite recent advances in face recognition using deep learning, severe accuracy drops are observed for large pose variations in unconstrained environments. Learning pose-invariant features is one solution, but needs expensively labeled large-scale data and carefully designed feature learning algorithms. In this work, we focus on frontalizing faces in the wild under various head poses, including extreme profile views. We propose a novel deep 3D Morphable Model (3DMM) conditioned Face Frontalization Generative Adversarial Network (GAN), termed as FF-GAN, to generate neutral head pose face image…
Georgia Institute of Technology（佐治亚理工学院）；Adobe Research（Adobe研究院）；Princeton University（普林斯顿大学）
Recently, there have been several promising methods to generate realistic imagery from deep convolutional networks. These methods sidestep the traditional computer graphics rendering pipeline and instead generate imagery at the pixel level by learning from large collections of photos (e.g. faces or bedrooms). However, these methods are of limited utility because it is difficult for a user to control what the network produces. In this paper, we propose a deep adversarial image synthesis architecture that is conditioned on sketched boundaries and sparse color strokes to generate realistic cars, …
University of Maryland College Park（马里兰大学帕克分校）；Tsinghua University（清华大学）
We propose a novel neural rendering pipeline, Hybrid Volumetric-Textural Rendering (HVTR), which synthesizes virtual human avatars from arbitrary poses efficiently and at high quality. First, we learn to encode articulated human motions on a dense UV manifold of the human body surface. To handle complicated motions (e.g., self-occlusions), we then leverage the encoded information on the UV manifold to construct a 3D volumetric representation based on a dynamic pose-conditioned neural radiance field. While this allows us to represent 3D geometry with changing topology, volumetric rendering is c…
Computer vision (CV) has achieved great success in interpreting semantic meanings from images, yet CV algorithms can be brittle for tasks with adverse vision conditions and the ones suffering from data/label pair limitation. One of this tasks is in-bed human pose estimation, which has significant values in many healthcare applications. In-bed pose monitoring in natural settings could involve complete darkness or full occlusion. Furthermore, the lack of publicly available in-bed pose datasets hinders the use of many successful pose estimation algorithms for this task. In this paper, we introduc…
University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；Purdue University（普渡大学）
Pose guided person image generation means to generate a photo-realistic person image conditioned on an input person image and a desired pose. This task requires spatial manipulation of the source image according to the target pose. However, the generative adversarial networks (GANs) widely used for image generation and translation rely on spatially local and translation equivariant operators, i.e., convolution, pooling and unpooling, which cannot handle large image deformation. This paper introduces a novel two-stream appearance transfer network (2s-ATN) to address this challenge. It is a mult…
Tianjin University（天津大学）；Wuhan University（武汉大学）；Tencent AI Lab（腾讯人工智能实验室）；Stony Brook University（石溪大学）
It is challenging to inpaint face images in the wild, due to the large variation of appearance, such as different poses, expressions and occlusions. A good inpainting algorithm should guarantee the realism of output, including the topological structure among eyes, nose and mouth, as well as the attribute consistency on pose, gender, ethnicity, expression, etc. This paper studies an effective deep learning based strategy to deal with these issues, which comprises of a facial landmark predicting subnet and an image inpainting subnet. Concretely, given partial observation, the landmark predictor …
Sun Yat-sen University（中山大学）；South China Normal University（华南师范大学）
Despite remarkable advances in image synthesis research, existing works often fail in manipulating images under the context of large geometric transformations. Synthesizing person images conditioned on arbitrary poses is one of the most representative examples where the generation quality largely relies on the capability of identifying and modeling arbitrary transformations on different body parts. Current generative models are often built on local convolutions and overlook the key challenges (e.g. heavy occlusions, different views or dramatic appearance changes) when distinct geometric change…
In this work, we present a new single-stage method for subject agnostic face swapping and identity transfer, named FaceDancer. We have two major contributions: Adaptive Feature Fusion Attention (AFFA) and Interpreted Feature Similarity Regularization (IFSR). The AFFA module is embedded in the decoder and adaptively learns to fuse attribute features and features conditioned on identity information without requiring any additional facial segmentation process. In IFSR, we leverage the intermediate features in an identity encoder to preserve important attributes such as head pose, facial expressio…
Indian Institute of Space Science and Technology（印度空间科学与技术学院）
Generating photorealistic images of human subjects in any unseen pose have crucial applications in generating a complete appearance model of the subject. However, from a computer vision perspective, this task becomes significantly challenging due to the inability of modelling the data distribution conditioned on pose. Existing works use a complicated pose transformation model with various additional features such as foreground segmentation, human body parsing etc. to achieve robustness that leads to computational overhead. In this work, we propose a simple yet effective pose transformation GAN…
Current 6D object pose methods consist of deep CNN models fully optimized for a single object but with its architecture standardized among objects with different shapes. In contrast to previous works, we explicitly exploit each object's distinct topological information i.e. 3D dense meshes in the pose estimation model, with an automated process and prior to any post-processing refinement stage. In order to achieve this, we propose a learning framework in which a Graph Convolutional Neural Network reconstructs a pose conditioned 3D mesh of the object. A robust estimation of the allocentric orie…
PMMD: A pose-guided multi-view multi-modal diffusion for person generation
Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学（深圳））；City University of Hong Kong, Hong Kong, China（香港城市大学）；Peking University, Beijing, China（北京大学）；Tsinghua University, Beijing, China（清华大学）
StableHand: Quality-Aware Flow Matching for World-Space Dual-Hand Motion Estimation from Egocentric Video
Mohamed bin Zayed University of Artificial Intelligence（莫扎德·本·泽德人工智能大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Imperial College London（伦敦帝国理工学院）
This work introduces a novel system for the generation of images that contain multiple classes of objects. Recent work in Generative Adversarial Networks have produced high quality images, but many focus on generating images of a single object or set of objects. Our system addresses the task of image generation conditioned on a list of desired classes to be included in a single image. This enables our system to generate images with any given combination of objects, all composed into a visually realistic natural image. The system learns the interrelationships of all classes represented in a dat…
Controllable Complex Human Motion Video Generation via Text-to-Skeleton Cascades
Department of Computer Science and Software Engineering, The University of Western Australia（计算机科学与软件工程系，西澳大学）；Munich Center for Machine Learning (MCML) and Technical University of Munich (TUM)（慕尼黑机器学习中心（MCML）和技术大学慕尼黑（TUM））；School of Information Technology, Murdoch University（信息科技学院，墨尔本大学）；Department of Electrical, Electronics and Computer Engineering, The University of Western Australia（电子、电子与计算机工程系，西澳大学）
Multi-modal 3D Pose and Shape Estimation with Computed Tomography
School of Computer Science, The University of Sydney, Sydney, NSW 2006, Australia（悉尼大学计算机科学学院）；School of Biomedical Engineering, The University of Sydney, Sydney, NSW 2006, Australia（悉尼大学生物医学工程学院）；Trauma Service, Westmead Hospital, Australia（西墨迪医院创伤服务）
