arXiv 2511.014632025-11-03cs.CVcs.AIcs.GR

HMVLM：基于MoE LoRA的人体运动-视觉-语言模型

HMVLM: Human Motion-Vision-Lanuage Model via MoE LoRA

Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）

Lei Hu, Yongjing Ye, Shihong Xia

备注：: 10 pages, 5figures. The Thirty-Ninth Annual Conference on Neural Information Processing Systems

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对人体运动与文本整合时的灾难性遗忘及姿态表示难题，提出基于MoE LoRA的HMVLM框架，通过门控网络动态分配专家权重并引入零专家保留预训练知识，同时采用身体部位分词化姿态表示，有效缓解知识遗忘并提升多任务性能。

摘要：

指令微调数据的扩展使基础语言模型在多种下游任务中展现出更好的指令遵循能力和卓越性能。语义丰富的3D人体运动正逐步与这些基础模型结合，以增强多模态理解和跨模态生成能力。然而，人体运动与文本之间的模态差异引发了关于整合过程中灾难性遗忘的未解问题。此外，开发保持异构下游任务泛化性且兼容自回归的姿态表示仍是关键技术障碍。为此，我们提出人体运动-视觉-语言模型（HMVLM），基于混合专家低秩自适应（MoE LoRA）策略的统一框架。该框架利用门控网络根据输入提示动态分配LoRA专家权重，实现多任务同步微调。为缓解指令微调中的灾难性遗忘，我们引入新颖的零专家，保留预训练参数以处理通用语言任务。在姿态表示上，通过将人体划分为不同关节组，实现特定身体部位的分词，提升表示的空间分辨率。实验表明，该方法有效缓解了指令微调中的知识遗忘，并在多样的人体运动下游任务中取得显著性能。

英文摘要：

The expansion of instruction-tuning data has enabled foundation language models to exhibit improved instruction adherence and superior performance across diverse downstream tasks. Semantically-rich 3D human motion is being progressively integrated with these foundation models to enhance multimodal understanding and cross-modal generation capabilities. However, the modality gap between human motion and text raises unresolved concerns about catastrophic forgetting during this integration. In addition, developing autoregressive-compatible pose representations that preserve generalizability across heterogeneous downstream tasks remains a critical technical barrier. To address these issues, we propose the Human Motion-Vision-Language Model (HMVLM), a unified framework based on the Mixture of Expert Low-Rank Adaption(MoE LoRA) strategy. The framework leverages the gating network to dynamically allocate LoRA expert weights based on the input prompt, enabling synchronized fine-tuning of multiple tasks. To mitigate catastrophic forgetting during instruction-tuning, we introduce a novel zero expert that preserves the pre-trained parameters for general linguistic tasks. For pose representation, we implement body-part-specific tokenization by partitioning the human body into different joint groups, enhancing the spatial resolution of the representation. Experiments show that our method effectively alleviates knowledge forgetting during instruction-tuning and achieves remarkable performance across diverse human motion downstream tasks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。