扩散模型在建模复杂数据分布方面展现了令人印象深刻的能力，并越来越多地应用于各种生成任务。在本工作中，我们提出了Pose Analysis by Diffusion Synthesis PADS，一个统一的生成建模框架，用于3D人体姿态分析。PADS首先通过无条件扩散合成学习一个任务无关的3D姿态先验，然后通过后验采样方案进行免训练适应，以适应广泛的姿态分析任务，包括3D姿态估计、去噪、补全等。通过将每个任务视为一个已知正向算子的逆问题，PADS在推理过程中注入任务特定的约束，同时保持姿态先验固定。这种即插即用的框架消除了对任务特定监督或重新训练的需要，提供了在多样化条件下灵活性和可扩展性。在不同基准上的广泛实验展示了其优于基于学习和优化的基线方法的优越性能，证明了该方法的有效性和泛化能力。
英文摘要：
Diffusion models have demonstrated impressive capabilities in modeling complex data distributions and are increasingly applied in various generative tasks. In this work, we propose Pose Analysis by Diffusion Synthesis PADS, a unified generative modeling framework for 3D human pose analysis. PADS first learns a task-agnostic 3D pose prior via unconditional diffusion synthesis and then performs training-free adaptation to a wide range of pose analysis tasks, including 3D pose estimation, denoising, completion, etc., through a posterior sampling scheme. By formulating each task as an inverse problem with a known forward operator, PADS injects task-specific constraints during inference while keeping the pose prior fixed. This plug-and-play framework removes the need for task-specific supervision or retraining, offering flexibility and scalability across diverse conditions. Extensive experiments on different benchmarks showcase the superior performance against both learning-based and optimization-based baselines, demonstrating the effectiveness and generalization capability of our method.
Fancy123: One Image to High-Quality 3D Mesh Generation via Plug-and-Play Deformation
Huazhong University of Science and Technology（华中科技大学）；Guangdong Intelligent Robotics Institute（广东智能机器人研究院）；South China University of Technology（华南理工大学）；Pazhou Laboratory（琶洲实验室）
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Tencent AI LAB（腾讯人工智能实验室）；Northeast Forestry University（东北林业大学）
The model-based gait recognition methods usually adopt the pedestrian walking postures to identify human beings. However, existing methods did not explicitly resolve the large intra-class variance of human pose due to camera views changing. In this paper, we propose to generate multi-view pose sequences for each single-view pose sample by learning full-rank transformation matrices via lower-upper generative adversarial network (LUGAN). By the prior of camera imaging, we derive that the spatial coordinates between cross-view poses satisfy a linear transformation of a full-rank matrix, thereby, …
The Chinese University of Hong Kong（香港中文大学）；Sensetime Group Ltd.（商汤集团有限公司）；Shanghai Jiao Tong University（上海交通大学）；Nanyang Technological University（南洋理工大学）
When analyzing human motion videos, the output jitters from existing pose estimators are highly-unbalanced with varied estimation errors across frames. Most frames in a video are relatively easy to estimate and only suffer from slight jitters. In contrast, for rarely seen or occluded actions, the estimated positions of multiple joints largely deviate from the ground truth values for a consecutive sequence of frames, rendering significant jitters on them. To tackle this problem, we propose to attach a dedicated temporal-only refinement network to existing pose estimators for jitter mitigation, …
HumanRefiner: Benchmarking Abnormal Human Generation and Refining with Coarse-to-fine Pose-Reversible Guidance
Sun Yat-sen University（中山大学）；Xi’an Jiaotong University（西安交通大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；United Arab Emirates University（阿联酋大学）
Recently, human pose estimation mainly focuses on how to design a more effective and better deep network structure as human features extractor, and most designed feature extraction networks only introduce the position of each anatomical keypoint to guide their training process. However, we found that some human anatomical keypoints kept their topology invariance, which can help to localize them more accurately when detecting the keypoints on the feature map. But to the best of our knowledge, there is no literature that has specifically studied it. Thus, in this paper, we present a novel 2D hum…
Plug-and-Play methods constitute a class of iterative algorithms for imaging problems where regularization is performed by an off-the-shelf denoiser. Although Plug-and-Play methods can lead to tremendous visual performance for various image problems, the few existing convergence guarantees are based on unrealistic (or suboptimal) hypotheses on the denoiser, or limited to strongly convex data terms. In this work, we propose a new type of Plug-and-Play methods, based on half-quadratic splitting, for which the denoiser is realized as a gradient descent step on a functional parameterized by a deep…
Removing optical and atmospheric blur from galaxy images significantly improves galaxy shape measurements for weak gravitational lensing and galaxy evolution studies. This ill-posed linear inverse problem is usually solved with deconvolution algorithms enhanced by regularisation priors or deep learning. We introduce a so-called "physics-informed deep learning" approach to the Point Spread Function (PSF) deconvolution problem in galaxy surveys. We apply algorithm unrolling and the Plug-and-Play technique to the Alternating Direction Method of Multipliers (ADMM), in which a neural network learns…
It's well-known that inverse problems are ill-posed and to solve them meaningfully, one has to employ regularization methods. Traditionally, popular regularization methods are the penalized Variational approaches. In recent years, the classical regularization approaches have been outclassed by the so-called plug-and-play (PnP) algorithms, which copy the proximal gradient minimization processes, such as ADMM or FISTA, but with any general denoiser. However, unlike the traditional proximal gradient methods, the theoretical underpinnings, convergence, and stability results have been insufficient …
In most state-of-the-art image restoration methods, the sum of a data-fidelity and a regularization term is optimized using an iterative algorithm such as ADMM (alternating direction method of multipliers). In recent years, the possibility of using denoisers for regularization has been explored in several works. A popular approach is to formally replace the proximal operator within the ADMM framework with some powerful denoiser. However, since most state-of-the-art denoisers cannot be posed as a proximal operator, one cannot guarantee the convergence of these so-called plug-and-play (PnP) algo…
PLUG: Revisiting Amodal Segmentation with Foundation Model and Hierarchical Focus
National Engineering Research Center of Visual Technology（国家视觉技术工程研究中心）；National Key Laboratory for Multimedia Information Processing（多媒体信息处理国家重点实验室）；School of Computer Science, Peking University（北京大学计算机学院）；AI Innovation Center, School of Computer Science, Peking University（北京大学计算机学院人工智能创新中心）；Meituan Inc.（美团公司）；National Biomedical Imaging Center, Peking University（北京大学生物医学成像中心）
Plug-and-Play (PnP) methods solve ill-posed inverse problems through iterative proximal algorithms by replacing a proximal operator by a denoising operation. When applied with deep neural network denoisers, these methods have shown state-of-the-art visual performance for image restoration problems. However, their theoretical convergence analysis is still incomplete. Most of the existing convergence results consider nonexpansive denoisers, which is non-realistic, or limit their analysis to strongly convex data-fidelity terms in the inverse problem to solve. Recently, it was proposed to train th…
Training state-of-the-art models for human pose estimation in videos requires datasets with annotations that are really hard and expensive to obtain. Although transformers have been recently utilized for body pose sequence modeling, related methods rely on pseudo-ground truth to augment the currently limited training data available for learning such models. In this paper, we introduce PoseBERT, a transformer module that is fully trained on 3D Motion Capture (MoCap) data via masked modeling. It is simple, generic and versatile, as it can be plugged on top of any image-based model to transform i…
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人社会研究院）；University of California, Irvine（加利福尼亚大学尔湾分校）
The end-to-end Human Mesh Recovery (HMR) approach has been successfully used for 3D body reconstruction. However, most HMR-based frameworks reconstruct human body by directly learning mesh parameters from images or videos, while lacking explicit guidance of 3D human pose in visual data. As a result, the generated mesh often exhibits incorrect pose for complex activities. To tackle this problem, we propose to exploit 3D pose to calibrate human mesh. Specifically, we develop two novel Pose Calibration frameworks, i.e., Serial PC-HMR and Parallel PC-HMR. By coupling advanced 3D pose estimators an…
University of Electronic Science Technology of China（电子科技大学）；Shenzhen University（深圳大学）；University of Surrey（萨里大学）
In this paper, we focus on the coordinate representation in human pose estimation. While being the standard choice, heatmap based representation has not been systematically investigated. We found that the process of coordinate decoding (i.e. transforming the predicted heatmaps to the coordinates) is surprisingly significant for human pose estimation performance, which nevertheless was not recognised before. In light of the discovered importance, we further probe the design limitations of the standard coordinate decoding method and propose a principled distribution-aware decoding method. Meanwh…
Training state-of-the-art models for human body pose and shape recovery from images or videos requires datasets with corresponding annotations that are really hard and expensive to obtain. Our goal in this paper is to study whether poses from 3D Motion Capture (MoCap) data can be used to improve image-based and video-based human mesh recovery methods. We find that fine-tune image-based models with synthetic renderings from MoCap data can increase their performance, by providing them with a wider variety of poses, textures and backgrounds. In fact, we show that simply fine-tuning the batch norm…
Megvii Inc（旷视科技）；University of Chinese Academy of Sciences (UCAS)（中国科学院大学）；Center for Research on Intelligent Perception and Computing (CRIPAC)（智能感知与计算研究中心）；National Laboratory of Pattern Recognition (NLPR)（模式识别国家重点实验室）；Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）
Most existing human pose estimation (HPE) methods exploit multi-scale information by fusing feature maps of four different spatial sizes, \ie $1/4$, $1/8$, $1/16$, and $1/32$ of the input image. There are two drawbacks of this strategy: 1) feature maps of different spatial sizes may be not well aligned spatially, which potentially hurts the accuracy of keypoint location; 2) these scales are fixed and inflexible, which may restrict the generalization ability over various human sizes. Towards these issues, we propose an adaptive dilated convolution (ADC). It can generate and fuse multi-scale fea…
Kinematics Modeling Network for Video-based Human Pose Estimation
Beijing University of Posts and Telecommunications（北京邮电大学）；Chinese Academy of Surveying and Mapping（中国测绘科学研究院）；Southwest Jiaotong University（西南交通大学）；Wuhan University（武汉大学）
Bayesian imaging using Plug & Play priors: when Langevin meets Tweedie
University of Oxford（牛津大学）；Institut Universitaire de France（法国大学研究院）；Heriot-Watt University（赫瑞瓦特大学）；Maxwell Institute for Mathematical Sciences（麦克斯韦数学科学研究所）
本文针对贝叶斯成像中 Plug & Play (PnP) 先验方法缺乏收敛证明的问题，提出了两种可证明收敛的算法 PnP-ULA 和 PnP-SGD，并建立了详细的收敛保证，适用于图像去模糊、修复和去噪等任务。
University of Electronic Science Technology of China（电子科技大学）；University of Surrey（萨里大学）
While being the de facto standard coordinate representation in human pose estimation, heatmap is never systematically investigated in the literature, to our best knowledge. This work fills this gap by studying the coordinate representation with a particular focus on the heatmap. Interestingly, we found that the process of decoding the predicted heatmaps into the final joint coordinates in the original image space is surprisingly significant for human pose estimation performance, which nevertheless was not recognised before. In light of the discovered importance, we further probe the design lim…
