弥合骨架-文本模态鸿沟：基于扩散模型的零样本骨架动作识别模态对齐
Bridging the Skeleton-Text Modality Gap: Diffusion-Powered Modality Alignment for Zero-shot Skeleton-based Action Recognition
AI总结：
本文提出基于扩散模型的骨架-文本对齐框架TDSM，通过将文本特征纳入反向扩散过程实现统一潜在空间匹配，并引入三重态扩散损失增强判别力，在ZSAR任务中显著超越现有方法。
摘要：
在零样本骨架动作识别（ZSAR）中，将骨架特征与动作标签的文本特征对齐对于准确预测未见动作至关重要。ZSAR面临弥合两种特征之间模态鸿沟的根本挑战，这严重限制了对未见动作的泛化。先前方法侧重于骨架与文本潜在空间之间的直接对齐，但这些空间之间的模态鸿沟阻碍了鲁棒的泛化学习。受扩散模型在多模态对齐（如文本到图像、文本到视频）中取得成功的启发，我们首次提出了用于ZSAR的基于扩散的骨架-文本对齐框架。我们的方法，用于骨架-文本匹配的三重态扩散（TDSM），侧重于扩散模型的交叉对齐能力而非其生成能力。具体而言，TDSM通过将文本特征纳入反向扩散过程，将骨架特征与文本提示对齐，其中骨架特征在文本指导下进行去噪，形成用于鲁棒匹配的统一骨架-文本潜在空间。为增强判别能力，我们引入了三重态扩散（TD）损失，鼓励我们的TDSM纠正骨架-文本匹配，同时将不同动作类别的匹配推开。我们的TDSM以2.36%到13.05%的显著大幅优势显著优于近期最先进的方法，通过有效的骨架-文本匹配展示了在零样本设置中卓越的准确性和可扩展性。
英文摘要：
In zero-shot skeleton-based action recognition (ZSAR), aligning skeleton features with the text features of action labels is essential for accurately predicting unseen actions. ZSAR faces a fundamental challenge in bridging the modality gap between the two-kind features, which severely limits generalization to unseen actions. Previous methods focus on direct alignment between skeleton and text latent spaces, but the modality gaps between these spaces hinder robust generalization learning. Motivated by the success of diffusion models in multi-modal alignment (e.g., text-to-image, text-to-video), we firstly present a diffusion-based skeleton-text alignment framework for ZSAR. Our approach, Triplet Diffusion for Skeleton-Text Matching (TDSM), focuses on cross-alignment power of diffusion models rather than their generative capability. Specifically, TDSM aligns skeleton features with text prompts by incorporating text features into the reverse diffusion process, where skeleton features are denoised under text guidance, forming a unified skeleton-text latent space for robust matching. To enhance discriminative power, we introduce a triplet diffusion (TD) loss that encourages our TDSM to correct skeleton-text matches while pushing them apart for different action classes. Our TDSM significantly outperforms very recent state-of-the-art methods with significantly large margins of 2.36%-point to 13.05%-point, demonstrating superior accuracy and scalability in zero-shot settings through effective skeleton-text matching.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
