CLUTCH：解锁野外文本条件手部运动建模的上下文化语言模型
CLUTCH: Contextualized Language model for Unlocking Text-Conditioned Hand motion modelling in the wild
AI总结：
CLUTCH通过引入3D-HIW数据集和SHIFT架构，提出基于LLM的手部动画系统，提升野外手部运动建模的保真度与泛化能力。
摘要：
手在日常生活中起着核心作用，但建模自然手部运动仍处于探索阶段。现有的针对文本到手部运动生成或手部动画描述的模型依赖于工作室捕捉的数据集，其动作和上下文有限，使其难以扩展到“野外”环境。此外，当前模型及其训练方案难以捕捉动画保真度与文本-运动对齐。为此，我们（1）引入“野外3D手部”（3D-HIW）数据集，包含32,000个3D手部运动序列和对齐的文本；（2）提出CLUTCH，一种基于LLM的手部动画系统，具有两个关键创新：（a）SHIFT，一种新的VQ-VAE架构用于标记手部运动；（b）几何细化阶段用于微调LLM。为构建3D-HIW，我们提出了一种结合视觉语言模型（VLM）和最新3D手部跟踪器的数据标注流程，并将其应用于大量第一人称动作视频，涵盖广泛场景。为全面捕捉野外运动，CLUTCH采用SHIFT，一种部分模态分解的VQ-VAE，提高了泛化能力和重建保真度。最后，为提高动画质量，我们引入了一个几何细化阶段，其中CLUTCH与重建损失共同监督，直接应用于解码的手部运动参数。实验显示在文本到运动和运动到文本任务中达到最先进的性能，建立了首个可扩展的野外手部运动建模基准。代码、数据和模型将被发布。
英文摘要：
Hands play a central role in daily life, yet modeling natural hand motions remains underexplored. Existing methods that tackle text-to-hand-motion generation or hand animation captioning rely on studio-captured datasets with limited actions and contexts, making them costly to scale to "in-the-wild" settings. Further, contemporary models and their training schemes struggle to capture animation fidelity with text-motion alignment. To address this, we (1) introduce '3D Hands in the Wild' (3D-HIW), a dataset of 32K 3D hand-motion sequences and aligned text, and (2) propose CLUTCH, an LLM-based hand animation system with two critical innovations: (a) SHIFT, a novel VQ-VAE architecture to tokenize hand motion, and (b) a geometric refinement stage to finetune the LLM. To build 3D-HIW, we propose a data annotation pipeline that combines vision-language models (VLMs) and state-of-the-art 3D hand trackers, and apply it to a large corpus of egocentric action videos covering a wide range of scenarios. To fully capture motion in-the-wild, CLUTCH employs SHIFT, a part-modality decomposed VQ-VAE, which improves generalization and reconstruction fidelity. Finally, to improve animation quality, we introduce a geometric refinement stage, where CLUTCH is co-supervised with a reconstruction loss applied directly to decoded hand motion parameters. Experiments demonstrate state-of-the-art performance on text-to-motion and motion-to-text tasks, establishing the first benchmark for scalable in-the-wild hand motion modelling. Code, data and models will be released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
