ChatAnyone：基于分层运动扩散模型的风格化实时人像视频生成
ChatAnyone: Stylized Real-time Portrait Video Generation with Hierarchical Motion Diffusion Model
AI总结：
提出ChatAnyone框架，通过分层运动扩散模型和显式手部控制，实现风格化实时上半身人像视频生成，支持4090 GPU上30fps的交互式聊天。
摘要：
实时交互式视频聊天人像正日益被认为是未来趋势，尤其是得益于文本和语音聊天技术取得的显著进展。然而，现有方法主要专注于头部运动的实时生成，却难以产生与这些头部动作相匹配的同步身体运动。此外，实现对说话风格和面部表情细微差别的精细控制仍然是一个挑战。为解决这些局限，我们引入了一个用于风格化实时人像视频生成的新框架，实现了从说话头部到上半身交互的富有表现力和灵活性的视频聊天。我们的方法包含以下两个阶段。第一阶段涉及高效的分层运动扩散模型，该模型基于音频输入同时考虑显式和隐式运动表示，能够生成具有风格控制以及头部和身体运动同步的多样化面部表情。第二阶段旨在生成包含手部手势的上半身运动的人像视频。我们将显式的手部控制信号注入生成器，以产生更细致的手部运动，并进一步执行面部细化，以增强人像视频的整体真实感和表现力。此外，我们的方法支持在4090 GPU上以最高30fps的帧率、最大512*768分辨率下高效且连续地生成上半身人像视频，支持实时交互式视频聊天。实验结果表明，我们的方法能够生成具有丰富表现力和自然上半身运动的人像视频。
英文摘要：
Real-time interactive video-chat portraits have been increasingly recognized as the future trend, particularly due to the remarkable progress made in text and voice chat technologies. However, existing methods primarily focus on real-time generation of head movements, but struggle to produce synchronized body motions that match these head actions. Additionally, achieving fine-grained control over the speaking style and nuances of facial expressions remains a challenge. To address these limitations, we introduce a novel framework for stylized real-time portrait video generation, enabling expressive and flexible video chat that extends from talking head to upper-body interaction. Our approach consists of the following two stages. The first stage involves efficient hierarchical motion diffusion models, that take both explicit and implicit motion representations into account based on audio inputs, which can generate a diverse range of facial expressions with stylistic control and synchronization between head and body movements. The second stage aims to generate portrait video featuring upper-body movements, including hand gestures. We inject explicit hand control signals into the generator to produce more detailed hand movements, and further perform face refinement to enhance the overall realism and expressiveness of the portrait video. Additionally, our approach supports efficient and continuous generation of upper-body portrait video in maximum 512 * 768 resolution at up to 30fps on 4090 GPU, supporting interactive video-chat in real-time. Experimental results demonstrate the capability of our approach to produce portrait videos with rich expressiveness and natural upper-body movements.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
