用于连续手语识别的去噪对比对齐
Denoising-Contrastive Alignment for Continuous Sign Language Recognition
AI总结：
针对连续手语识别中跨模态对齐忽视文本语法与全局时序上下文的问题，提出 DCA 范式，结合对比实例对齐、视频引导的 gloss 去噪及梯度调制以增强视频表示。
摘要：
连续手语识别（CSLR）旨在识别未裁剪手语视频中的手语，并将其转换为文本手语 gloss。CSLR 的一个关键挑战是在视频与 gloss 序列之间实现有效的跨模态对齐，以增强视频表示。然而，当前的跨模态对齐范式往往忽视文本语法在指导视频表示学习全局时间上下文方面的作用，从而对识别性能产生不利影响。为解决这一局限，我们提出一种去噪对比对齐（Denoising-Contrastive Alignment，DCA）范式。DCA 创造性地利用文本语法，通过两种互补方法增强视频表示：从判别视角建模手语与 gloss 之间的实例对应关系，并从生成视角对齐它们的全局上下文。具体而言，DCA 使用对比损失实现手语与 gloss 之间灵活的实例级对应。在此基础上，DCA 通过在视频表示引导下对含噪 gloss 表示进行去噪，建模视频序列与 gloss 序列之间的全局上下文对齐。此外，DCA 引入梯度调制来优化对齐梯度和识别梯度，确保学习过程更有效。通过融合逐 gloss 知识与全局上下文知识，DCA 显著增强了面向 CSLR 任务的视频表示。在多个公开基准上的实验结果验证了 DCA 的有效性，并证实了其增强视频表示的可行性。
英文摘要：
Continuous sign language recognition (CSLR) aims to recognize signs in untrimmed sign language videos to textual glosses. A key challenge of CSLR is achieving effective cross-modality alignment between video and gloss sequences to enhance video representation. However, current cross-modality alignment paradigms often neglect the role of textual grammar to guide the video representation in learning global temporal context, which adversely affects recognition performance. To tackle this limitation, we propose a Denoising-Contrastive Alignment (DCA) paradigm. DCA creatively leverages textual grammar to enhance video representations through two complementary approaches: modeling the instance correspondence between signs and glosses from a discrimination perspective and aligning their global context from a generative perspective. Specifically, DCA accomplishes flexible instance-level correspondence between signs and glosses using a contrastive loss. Building on this, DCA models global context alignment between the video and gloss sequences by denoising the gloss representation from noise, guided by video representation. Additionally, DCA introduces gradient modulation to optimize the alignment and recognition gradients, ensuring a more effective learning process. By integrating gloss-wise and global context knowledge, DCA significantly enhances video representations for CSLR tasks. Experimental results across public benchmarks validate the effectiveness of DCA and confirm its video representation enhancement feasibility.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
