GHOST：基于开放词汇场景与文本上下文的基础人类动作生成
GHOST: Grounded Human Motion Generation with Open Vocabulary Scene-and-Text Contexts
AI总结：
本文提出GHOST方法，通过集成开放词汇场景编码器并采用知识蒸馏与微调策略，结合两种正则化损失，在HUMANISE数据集上将目标物体距离指标降低30%，显著提升了基于文本与场景的人类动作生成与定位精度。
摘要：
我们的3D环境与描述这些环境的语言之间的联系，非常适合用于在上下文中定位和生成人类动作，但存在一个问题。多模态引入的复杂性使得用一组固定的描述符来捕捉这种联系极具挑战性。具体而言，文献中常采用封闭词汇场景编码器，这类编码器需要从头学习文本与场景的关联，往往导致动作定位不准确。在本文中，我们提出了一种将开放词汇场景编码器集成到架构中的方法，从而在文本与场景之间建立了稳健的联系。我们的两步法首先通过从现有的开放词汇语义图像分割模型进行知识蒸馏来预训练场景编码器，确保共享的文本-场景特征空间。随后，对场景编码器进行微调以实现条件动作生成，并引入了两种新的正则化损失，用于回归目标物体的类别和大小。在HUMANISE数据集上，与先前的最先进基准模型相比，我们的方法在目标物体距离指标上实现了高达30%的降低。通过使用我们框架的三种实现方案进行的评估以及一项感知研究，证明了这一改进。此外，我们的方法被设计为能够无缝兼容未来提供像素级文本对齐特征以供蒸馏的2D分割方法。
英文摘要：
The connection between our 3D surroundings and the descriptive language that characterizes them would be well-suited for localizing and generating human motion in context but for one problem. The complexity introduced by multiple modalities makes capturing this connection challenging with a fixed set of descriptors. Specifically, closed vocabulary scene encoders, which require learning text-scene associations from scratch, have been favored in the literature, often resulting in inaccurate motion grounding. In this paper, we propose a method that integrates an open vocabulary scene encoder into the architecture, establishing a robust connection between text and scene. Our two-step approach starts with pretraining the scene encoder through knowledge distillation from an existing open vocabulary semantic image segmentation model, ensuring a shared text-scene feature space. Subsequently, the scene encoder is fine-tuned for conditional motion generation, incorporating two novel regularization losses that regress the category and size of the goal object. Our methodology achieves up to a 30% reduction in the goal object distance metric compared to the prior state-of-the-art baseline model on the HUMANISE dataset. This improvement is demonstrated through evaluations conducted using three implementations of our framework and a perceptual study. Additionally, our method is designed to seamlessly accommodate future 2D segmentation methods that provide per-pixel text-aligned features for distillation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
