arXiv 2603.180032026-05-21cs.CV

通过可微渲染和大语言模型实现通用骨架理解

Universal Skeleton Understanding via Differentiable Rendering and MLLMs

State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School, China（人工智能通用基础理论国家重点实验室，北京大学深圳研究生院，中国）；Tencent（腾讯）；Nanjing University of Aeronautics（南京航空航天大学）

Ziyi Wang, Peiming Li, Xinshun Wang, Yang Tang, Kai-Kuang Ma, Mengyuan Liu

备注：: Accepted by ICML 2026

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出SkeletonLLM，通过可微渲染将任意骨架序列转换为大语言模型的视觉模态，实现通用骨架理解，同时引入协同训练策略提升推理能力，展示了在开放词汇动作识别中的强泛化能力，并扩展到异构骨架格式的运动描述和问答任务。

摘要：

多模态大语言模型（MLLMs）在视觉-语言推理方面表现出色，但无法处理结构化非视觉数据如人体骨架。现有方法要么将骨架动力学压缩成有损特征向量以进行文本对齐，要么将运动量化为离散标记，但这些方法在异构骨架格式上泛化能力较差。我们提出了SkeletonLLM，通过将任意骨架序列转换为MLLM的本机视觉模态实现通用骨架理解。其核心是DrAction，一种可微、格式无关的渲染器，将骨骼运动学转换为紧凑的图像序列。由于整个流程是端到端可微的，MLLM的梯度可以直接引导渲染以生成任务相关信息的视觉标记。为进一步增强推理能力，我们引入了协同训练策略：因果推理蒸馏将结构化的逐步推理从教师模型转移过来，而判别微调则增强可混淆动作之间的决策边界。SkeletonLLM在开放词汇动作识别中表现出强泛化能力，其学习的推理能力自然扩展到异构骨架格式的运动描述和问答任务——表明了将MLLM应用于非本机模态的可行路径。代码：https://github.com/wangzy01/SkeletonLLM。

英文摘要：

Multimodal large language models (MLLMs) exhibit strong visual-language reasoning, yet cannot process structured, non-visual data such as human skeletons. Existing methods either compress skeleton dynamics into lossy feature vectors for text alignment, or quantize motion into discrete tokens that generalize poorly across heterogeneous skeleton formats. We present SkeletonLLM, which achieves universal skeleton understanding by translating arbitrary skeleton sequences into the MLLM's native visual modality. At its core is DrAction, a differentiable, format-agnostic renderer that converts skeletal kinematics into compact image sequences. Because the pipeline is end-to-end differentiable, MLLM gradients can directly guide the rendering to produce task-informative visual tokens. To further enhance reasoning capabilities, we introduce a cooperative training strategy: Causal Reasoning Distillation transfers structured, step-by-step reasoning from a teacher model, while Discriminative Finetuning sharpens decision boundaries between confusable actions. SkeletonLLM demonstrates strong generalization \revise{in open-vocabulary action recognition, while its learned reasoning capabilities naturally extend to motion captioning and question answering across heterogeneous skeleton formats} -- suggesting a viable path for applying MLLMs to non-native modalities. Code: https://github.com/wangzy01/SkeletonLLM.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。