Lang2Motion: 通过联合嵌入空间连接语言与运动
Lang2Motion: Bridging Language and Motion through Joint Embedding Spaces
AI总结：
Lang2Motion通过联合嵌入空间连接语言与运动，利用文本描述和视觉化监督生成高精度的轨迹，实现跨领域运动迁移和风格编辑。
摘要：
我们提出了Lang2Motion，一种通过对齐运动流形与联合嵌入空间来生成语言引导的点轨迹的框架。与以往专注于人类运动或视频合成的工作不同，我们通过点跟踪从现实世界视频中提取运动，为任意物体生成显式轨迹。我们的基于Transformer的自编码器通过双重监督学习轨迹表示：文本运动描述和渲染的轨迹可视化，两者均通过CLIP冻结编码器映射。Lang2Motion在文本到轨迹检索中实现了34.2%的Recall@1，比基于视频的方法高12.5个百分点，并在运动准确性方面比视频生成基线提高了33-52%（12.4 ADE vs 18.3-25.3）。我们尽管只在多样化的物体运动上训练，却在人类动作识别上实现了88.3%的Top-1准确率，展示了在运动领域之间的有效迁移。Lang2Motion通过CLIP对齐的轨迹表示支持风格迁移、语义插值和潜在空间编辑。
英文摘要：
We present Lang2Motion, a framework for language-guided point trajectory generation by aligning motion manifolds with joint embedding spaces. Unlike prior work focusing on human motion or video synthesis, we generate explicit trajectories for arbitrary objects using motion extracted from real-world videos via point tracking. Our transformer-based auto-encoder learns trajectory representations through dual supervision: textual motion descriptions and rendered trajectory visualizations, both mapped through CLIP's frozen encoders. Lang2Motion achieves 34.2% Recall@1 on text-to-trajectory retrieval, outperforming video-based methods by 12.5 points, and improves motion accuracy by 33-52% (12.4 ADE vs 18.3-25.3) compared to video generation baselines. We demonstrate 88.3% Top-1 accuracy on human action recognition despite training only on diverse object motions, showing effective transfer across motion domains. Lang2Motion supports style transfer, semantic interpolation, and latent-space editing through CLIP-aligned trajectory representations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
