ViPo-MLLM：用于无注释手语翻译的视觉-姿势多模态大语言模型
ViPo-MLLM: Visual-Pose Multimodal LLM for Gloss-Free Sign Language Translation
AI总结：
研究尝试无注释手语翻译，提出ViPo-MLLM框架结合时空RGB和人体姿势特征，用专用编码器与交叉模态注意力，经结构化提示和训练后由大语言模型处理。该模型在数据集取得新成果，验证了机制有效性。
摘要：
无注释手语翻译（SLT）将手语视频翻译成口语句子，无需注释，但需要对手部、身体和面部线索进行细粒度建模。现有方法常使用单模态或弱融合特征，限制了性能。我们提出了ViPo-MLLM，一个整合时空RGB和人体姿势特征的框架。专用编码器对模态内动态进行建模，交叉模态注意力捕捉长程依赖。融合表示由结构化提示调节，并由通过对比和语言建模目标训练的大语言模型处理。所提出的模型在PHOENIX14T和CSL-Daily数据集上进行了评估，并在两个数据集上取得了新的最优结果。此外，与基于注释的识别方法相比，ViPo-MLLM模型具有竞争力的性能，证实了所提出的姿势线索和交叉模态注意力机制的有效性。
英文摘要：
Gloss-free Sign Language Translation (SLT) translates sign language videos into spoken-language sentences without gloss annotations, avoiding costly labeling but requiring fine-grained modeling of hands, body, and facial cues. Existing methods often use single-modality or weakly fused features, limiting performance. We propose ViPo-MLLM, a framework that integrates spatio-temporal RGB and human pose features. Dedicated encoders model intra-modal dynamics and cross-modal attention captures long-range dependencies. The fused representation is conditioned with a structured prompt and processed by an LLM trained with contrastive and language modeling objectives. The proposed model was evaluated on the PHOENIX14T and CSL-Daily datasets and achieved new state-of-the-art results on both datasets. Moreover, the ViPo-MLLM model attained competitive performance compared to gloss-based recognition approaches, confirming the effectiveness of the proposed pose cues and cross-modal attention mechanisms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
