Text-to-Motion Retrieval：面向人体运动数据与自然语言的联合理解
Text-to-Motion Retrieval: Towards Joint Understanding of Human Motion Data and Natural Language
AI总结：
本文提出文本到运动检索新任务，构建BERT、CLIP等基线并提出基于划分式时空注意力的MoT方法，在两个数据集上建立评估协议，推动人体运动与自然语言的联合理解。
摘要：
随着姿态估计方法的最新进展，人体运动能够以3D骨架序列的形式从普通视频中提取出来。尽管存在广阔的应用前景，但对大量此类时空骨架数据进行高效、有效的基于内容的访问仍然是一个具有挑战性的问题。本文提出了一种新颖的基于内容的text-to-motion retrieval（文本到运动检索）任务，旨在根据指定的自然语言文本描述检索相关运动。为了给这一全新任务定义基准，我们采用BERT和CLIP语言表示来编码文本模态，并用成熟的时空模型来编码运动模态。我们还提出了基于transformer的方法Motion Transformer (MoT)，该方法采用划分式时空注意力，有效聚合空间和时间维度上的不同骨架关节。受text-to-image/video matching（文本到图像/视频匹配）领域最新进展的启发，我们对两种广泛使用的度量学习损失函数进行了实验。最后，我们针对近期提出的KIT Motion-Language和HumanML3D两个数据集，通过定义评估检索运动质量的定性指标，建立了一套通用评估协议。复现我们结果的代码可在https://github.com/mesnico/text-to-motion-retrieval获取。
英文摘要：
Due to recent advances in pose-estimation methods, human motion can be extracted from a common video in the form of 3D skeleton sequences. Despite wonderful application opportunities, effective and efficient content-based access to large volumes of such spatio-temporal skeleton data still remains a challenging problem. In this paper, we propose a novel content-based text-to-motion retrieval task, which aims at retrieving relevant motions based on a specified natural-language textual description. To define baselines for this uncharted task, we employ the BERT and CLIP language representations to encode the text modality and successful spatio-temporal models to encode the motion modality. We additionally introduce our transformer-based approach, called Motion Transformer (MoT), which employs divided space-time attention to effectively aggregate the different skeleton joints in space and time. Inspired by the recent progress in text-to-image/video matching, we experiment with two widely-adopted metric-learning loss functions. Finally, we set up a common evaluation protocol by defining qualitative metrics for assessing the quality of the retrieved motions, targeting the two recently-introduced KIT Motion-Language and HumanML3D datasets. The code for reproducing our results is available at https://github.com/mesnico/text-to-motion-retrieval.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
