无需级联的 Mandarin 视觉语音识别 via 语义引导的跨表征对齐
Cascade-Free Mandarin Visual Speech Recognition via Semantic-Guided Cross-Representation Alignment
AI总结：
本文提出一种无需级联的多任务学习架构，通过整合音素和viseme等中间表示，利用语义引导的局部对比损失对齐特征，提升识别性能并减少误差累积。
摘要：
中文 Mandarin 视觉语音识别（VSR）近年来有所进展，但仍落后于非声调语言如英语的性能。主要挑战来自 Mandarin 的声调性质，限制了传统序列到序列建模方法的有效性。为缓解这一问题，现有 Chinese VSR 系统通常在级联架构中整合中间表示，如拼音，以提高识别准确性。尽管有益，这些级联设计在推理过程中后续阶段依赖前一阶段的输出，导致误差累积和推理延迟增加。为解决这些限制，我们提出了一种基于多任务学习的无需级联架构，联合整合多个中间表示，包括音素和 viseme，以更好地利用上下文信息。所提出的语义引导的局部对比损失在时间上对齐特征，使推理过程中能够按需激活，从而在推理效率和性能之间取得平衡，同时减轻投影和重新嵌入导致的误差累积。在公开可用数据集上的实验表明，我们的方法实现了更优的识别性能。
英文摘要：
Chinese mandarin visual speech recognition (VSR) is a task that has advanced in recent years, yet still lags behind the performance on non-tonal languages such as English. One primary challenge arises from the tonal nature of Mandarin, which limits the effectiveness of conventional sequence-to-sequence modeling approaches. To alleviate this issue, existing Chinese VSR systems commonly incorporate intermediate representations, most notably pinyin, within cascade architectures to enhance recognition accuracy. While beneficial, in these cascaded designs, the subsequent stage during inference depends on the output of the preceding stage, leading to error accumulation and increased inference latency. To address these limitations, we propose a cascade-free architecture based on multitask learning that jointly integrates multiple intermediate representations, including phoneme and viseme, to better exploit contextual information. The proposed semantic-guided local contrastive loss temporally aligns the features, enabling on-demand activation during inference, thereby providing a trade-off between inference efficiency and performance while mitigating error accumulation caused by projection and re-embedding. Experiments conducted on publicly available datasets demonstrate that our method achieves superior recognition performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
