arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.09883cs.CV
收藏

DISPLAY:通过稀疏运动引导和多任务辅助实现可直接控制的人机交互视频生成

DISPLAY: Directable Human-Object Interaction Video Generation via Sparse Motion Guidance and Multi-Task Auxiliary

Baidu Inc.(百度公司)

Jiazhi Guan, Quanwei Yang, Luying Huang, Junhao Liang, Borong Liang, Haocheng Feng, Wei He, Kaisiyuan Wang, Hang Zhou, Jingdong Wang

AI总结:

DISPLAY通过稀疏运动引导和多任务辅助训练,实现高保真且可控的人机交互视频生成。

摘要:

以人为核心视频生成技术已取得显著进展,但现有方法难以生成可控且物理一致的人机交互(HOI)视频。现有方法依赖密集控制信号、模板视频或精心设计的文本提示,这限制了灵活性和对新对象的泛化能力。我们引入了一个名为DISPLAY的框架,其指导原则是稀疏运动引导,仅包含手腕关节坐标和不依赖形状的物体边界框。这种轻量级引导缓解了人与物体表示之间的不平衡,并使用户能够直观地进行控制。为了在如此稀疏的条件下提高保真度,我们提出了一个物体应力注意力机制,以增强物体的鲁棒性。为了解决高质量HOI数据稀缺的问题,我们进一步开发了多任务辅助训练策略,并配备专门的数据整理流程,使模型能够从可靠的HOI样本和辅助任务中受益。全面的实验表明,我们的方法在多样化的任务中实现了高保真度、可控的HOI生成。项目页面可在https://mumuwei.github.io/DISPLAY/上找到。

英文摘要:

Human-centric video generation has advanced rapidly, yet existing methods struggle to produce controllable and physically consistent Human-Object Interaction (HOI) videos. Existing works rely on dense control signals, template videos, or carefully crafted text prompts, which limit flexibility and generalization to novel objects. We introduce a framework, namely DISPLAY, guided by Sparse Motion Guidance, composed only of wrist joint coordinates and a shape-agnostic object bounding box. This lightweight guidance alleviates the imbalance between human and object representations and enables intuitive user control. To enhance fidelity under such sparse conditions, we propose an Object-Stressed Attention mechanism that improves object robustness. To address the scarcity of high-quality HOI data, we further develop a Multi-Task Auxiliary Training strategy with a dedicated data curation pipeline, allowing the model to benefit from both reliable HOI samples and auxiliary tasks. Comprehensive experiments show that our method achieves high-fidelity, controllable HOI generation across diverse tasks. The project page can be found at \href{https://mumuwei.github.io/DISPLAY/}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑