通过运动学短语弥合人体运动与动作语义之间的鸿沟
Bridging the Gap between Human Motion and Action Semantics via Kinematic Phrases
AI总结:
针对人体运动与动作语义间模态鸿沟导致映射可靠性低、评估指标失效的问题,提出具备可解释性的运动学短语KP,构建运动理解系统与白盒生成基准KPG,实验表现优于现有方法。
摘要:
运动理解旨在建立运动与动作语义之间可靠的映射关系,然而这是一个颇具挑战性的多对多问题。一个抽象的动作语义(例如“向前走”)可以通过感知上截然不同的运动来传达(比如手臂抬起走路或摆臂走路)。反之,一个运动根据其所处情境和意图的不同,也可能承载不同的语义。这使得二者之间难以构建精妙的映射。过往研究采用直接映射范式,可靠性有限。此外,当前的自动指标无法对运动与动作语义之间的一致性提供可靠评估。我们认为这些问题的根源在于两种模态之间存在显著鸿沟。为缩小这一鸿沟,我们提出了运动学短语(Kinematic Phrases,简称KP),它承载人体运动的客观运动学事实,具备适当的抽象性、可解释性与通用性。基于KP,我们能够统一运动知识库并构建运动理解系统。同时,KP可以从运动自动转换为文本描述,不存在主观偏差,由此催生了运动学提示生成(Kinematic Prompt Generation,简称KPG)这一新型白盒运动生成基准。在大量实验中,我们的方法展现出优于其他方法的性能。我们的项目可在https://foruck.github.io/KP/获取。
英文摘要:
Motion understanding aims to establish a reliable mapping between motion and action semantics, while it is a challenging many-to-many problem. An abstract action semantic (i.e., walk forwards) could be conveyed by perceptually diverse motions (walking with arms up or swinging). In contrast, a motion could carry different semantics w.r.t. its context and intention. This makes an elegant mapping between them difficult. Previous attempts adopted direct-mapping paradigms with limited reliability. Also, current automatic metrics fail to provide reliable assessments of the consistency between motions and action semantics. We identify the source of these problems as the significant gap between the two modalities. To alleviate this gap, we propose Kinematic Phrases (KP) that take the objective kinematic facts of human motion with proper abstraction, interpretability, and generality. Based on KP, we can unify a motion knowledge base and build a motion understanding system. Meanwhile, KP can be automatically converted from motions to text descriptions with no subjective bias, inspiring Kinematic Prompt Generation (KPG) as a novel white-box motion generation benchmark. In extensive experiments, our approach shows superiority over other methods. Our project is available at https://foruck.github.io/KP/.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。