arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.13657cs.CVcs.AI
收藏

MLP:未修剪3D人体运动中时序句子定位的运动标签先验

MLP: Motion Label Prior for Temporal Sentence Localization in Untrimmed 3D Human Motions

College of Computer Science, Sichuan University(四川大学计算机学院)

Sheng Yan, Mengyuan Liu, Yong Wang, Yang Liu, Chen Chen, Hong Liu

AI总结:

本文针对3D人体运动时序句子定位问题,提出两种标签先验辅助训练方案以解决数据语义模糊导致的预测粗糙问题,其MLP模型在BABEL和HumanML3D数据集上高IoU指标均超越先前工作。

摘要:

在本文中,我们探讨了人体运动中时序句子定位(TSLM)这一未探索的问题,旨在从3D人体运动中定位与文本查询在语义上对应的目标时刻。考虑到3D人体运动是使用专门的动捕设备捕获的,仅包含少数关节的运动缺乏诸如物体和光照等复杂场景信息。由于这一特征,运动数据的上下文丰富度较低且帧间存在语义模糊性,这限制了当前扩展到TSLM的视频定位框架的预测精度仅停留在粗糙水平。为了改进这一点,我们设计了两种新颖的标签先验辅助训练方案:一种嵌入前景和背景的先验知识以突出目标时刻的定位机会,另一种在恢复训练期间迫使原本粗糙的预测与从翻转的起始/结束先验标签序列获得的更准确预测相重叠。我们表明,将标签先验知识注入模型对于提高高IoU下的性能至关重要。在我们构建的TSLM基准中,我们称为MLP的模型在BABEL数据集上IoU@0.7达到了44.13的召回率,在HumanML3D(Restore)上达到71.17,超越了先前的工作。最后,我们展示了我们的方法在语料库级别时刻检索中的潜力。我们的源代码已在https://github.com/eanson023/mlp公开。

英文摘要:

In this paper, we address the unexplored question of temporal sentence localization in human motions (TSLM), aiming to locate a target moment from a 3D human motion that semantically corresponds to a text query. Considering that 3D human motions are captured using specialized motion capture devices, motions with only a few joints lack complex scene information like objects and lighting. Due to this character, motion data has low contextual richness and semantic ambiguity between frames, which limits the accuracy of predictions made by current video localization frameworks extended to TSLM to only a rough level. To refine this, we devise two novel label-prior-assisted training schemes: one embed prior knowledge of foreground and background to highlight the localization chances of target moments, and the other forces the originally rough predictions to overlap with the more accurate predictions obtained from the flipped start/end prior label sequences during recovery training. We show that injecting label-prior knowledge into the model is crucial for improving performance at high IoU. In our constructed TSLM benchmark, our model termed MLP achieves a recall of 44.13 at IoU@0.7 on the BABEL dataset and 71.17 on HumanML3D (Restore), outperforming prior works. Finally, we showcase the potential of our approach in corpus-level moment retrieval. Our source code is openly accessible at https://github.com/eanson023/mlp.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑