arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.12370cs.CV
收藏

LLaMo:通过连续自回归令牌扩展预训练语言模型,实现统一的运动理解和生成

LLaMo: Scaling Pretrained Language Models for Unified Motion Understanding and Generation with Continuous Autoregressive Tokens

Brown University(布朗大学);Meta

Zekun Li, Sizhe An, Chengcheng Tang, Chuan Guo, Ivan Shugurov, Linguang Zhang, Amy Zhao, Srinath Sridhar, Lingling Tao, Abhay Mittal

AI总结:

LLaMo通过模态特定的Transformer混合架构扩展预训练语言模型,解决运动-语言生成和理解的统一问题,实现高保真文本到运动生成和运动到文本描述。

摘要:

近年来,大模型的进步在统一多模态生成和理解方面取得了显著进展。然而,统一运动-语言生成和理解的模型开发仍处于探索阶段。现有方法通常在配对的运动-文本数据上微调大型语言模型(LLMs),由于可用文本-运动配对有限,这会导致语言能力的灾难性遗忘。此外,先前方法通常通过量化将运动转换为离散表示,以与语言模型集成,引入了显著的抖动伪影。为了解决这些挑战,我们提出了LLaMo,一个统一的框架,通过模态特定的Transformer混合架构(MoT)扩展预训练LLMs。这种设计本质上保留了基础模型的语言理解能力,同时实现了可扩展的多模态适应。我们通过因果连续潜在空间编码人类运动,并通过轻量级流匹配头在解码器-only骨干中保持下一个令牌预测范式,从而实现实时(>30 FPS)的流式运动生成。利用预训练LLMs的全面语言理解和大规模运动-文本预训练,我们的实验表明,LLaMo在一般场景中实现了高保真文本到运动生成和运动到文本描述,尤其是在零样本运动生成方面,标志着向通用统一运动-语言大模型迈出重要一步。

英文摘要:

Recent progress in large models has led to significant advances in unified multimodal generation and understanding. However, the development of models that unify motion-language generation and understanding remains largely underexplored. Existing approaches often fine-tune large language models (LLMs) on paired motion-text data, which can result in catastrophic forgetting of linguistic capabilities due to the limited scale of available text-motion pairs. Furthermore, prior methods typically convert motion into discrete representations via quantization to integrate with language models, introducing substantial jitter artifacts from discrete tokenization. To address these challenges, we propose LLaMo, a unified framework that extends pretrained LLMs through a modality-specific Mixture-of-Transformers (MoT) architecture. This design inherently preserves the language understanding of the base model while enabling scalable multimodal adaptation. We encode human motion into a causal continuous latent space and maintain the next-token prediction paradigm in the decoder-only backbone through a lightweight flow-matching head, allowing for streaming motion generation in real-time (>30 FPS). Leveraging the comprehensive language understanding of pretrained LLMs and large-scale motion-text pretraining, our experiments demonstrate that LLaMo achieves high-fidelity text-to-motion generation and motion-to-text captioning in general settings, especially zero-shot motion generation, marking a significant step towards a general unified motion-language large model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑