HHMR：通过增强图扩散模型的多模态可控性实现整体手部网格恢复
HHMR: Holistic Hand Mesh Recovery by Enhancing the Multimodal Controllability of Graph Diffusion Models
AI总结：
本文提出HHMR框架，通过基于图卷积和注意力机制的一体化扩散模型，实现手部网格生成、修复、重建和拟合等多任务，并利用条件对齐梯度引导提升精度，在多项任务中超越现有方法。
摘要：
近年来，生成与重建范式的深度融合已成为一种趋势。本文扩展了可控生成模型的能力，以实现更全面的手部网格恢复任务：在单一框架内实现直接手部网格生成、图像修复、重建和拟合，我们将其命名为整体手部网格恢复（HHMR）。我们的关键观察是，不同类型的手部网格恢复任务可以通过一个具有强大多模态可控性的单一生成模型来实现，在此框架中，实现不同任务仅需提供不同的信号作为条件。为实现这一目标，我们提出了一个基于图卷积和注意力机制的一体化扩散框架，用于整体手部网格恢复。为了在确保多模态控制信号解耦的同时实现强大的控制生成能力，我们将不同模态映射到共享特征空间，并在模态和特征层面应用跨尺度随机掩码。通过这种方式，在学习手部先验时可以充分利用不同模态之间的相关性。此外，我们提出了条件对齐梯度引导，以增强生成模型与控制信号的对齐，从而显著提高手部网格重建和拟合的准确性。实验表明，我们的新框架能够同时实现多种手部网格恢复任务，并在不同任务中优于现有方法，为包括手势识别、姿态生成、网格编辑等在内的后续下游应用提供了更多可能性。
英文摘要：
Recent years have witnessed a trend of the deep integration of the generation and reconstruction paradigms. In this paper, we extend the ability of controllable generative models for a more comprehensive hand mesh recovery task: direct hand mesh generation, inpainting, reconstruction, and fitting in a single framework, which we name as Holistic Hand Mesh Recovery (HHMR). Our key observation is that different kinds of hand mesh recovery tasks can be achieved by a single generative model with strong multimodal controllability, and in such a framework, realizing different tasks only requires giving different signals as conditions. To achieve this goal, we propose an all-in-one diffusion framework based on graph convolution and attention mechanisms for holistic hand mesh recovery. In order to achieve strong control generation capability while ensuring the decoupling of multimodal control signals, we map different modalities to a shared feature space and apply cross-scale random masking in both modality and feature levels. In this way, the correlation between different modalities can be fully exploited during the learning of hand priors. Furthermore, we propose Condition-aligned Gradient Guidance to enhance the alignment of the generated model with the control signals, which significantly improves the accuracy of the hand mesh reconstruction and fitting. Experiments show that our novel framework can realize multiple hand mesh recovery tasks simultaneously and outperform the existing methods in different tasks, which provides more possibilities for subsequent downstream applications including gesture recognition, pose generation, mesh editing, and so on.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
