面向提示语音识别的计算与参数高效多模态融合Transformer
Computation and Parameter Efficient Multi-Modal Fusion Transformer for Cued Speech Recognition
AI总结：
针对自动提示语音识别中多模态融合的全局依赖与效率问题，提出Token重要性感知注意力机制与轻量门控投影，构建计算与参数高效的EcoCued模型，在全部数据集上取得最优性能。
摘要：
提示语音（CS）是一种纯视觉编码方法，供听力受损人群使用，它将唇读与几种特定的手形相结合，使口语变得可见。自动提示语音识别（ACSR）旨在将语音的视觉线索转录为文本，这有助于听力受损人群进行有效沟通。CS的视觉信息包含唇读和手部提示，因此它们的融合在ACSR中起着重要作用。然而，以往大多数融合方法难以捕捉多模态CS数据长序列输入中的全局依赖性。因此，这些方法通常无法学习到有助于融合的有效跨模态关系。近年来，基于注意力的Transformer已成为在多模态融合中捕捉长序列全局依赖性的流行思路，但现有的多模态融合Transformer在ACSR任务中既存在识别精度不高的问题，也存在计算效率低下的问题。为解决这些问题，我们开发了一种新颖的计算与参数高效多模态融合Transformer，提出了一种新颖的Token重要性感知注意力机制（TIAA），其中定义了令牌利用率（TUR）来从多模态流中选择重要令牌。更具体地说，TIAA首先对每个模态的所有令牌建模模态特定的细粒度时间依赖性，然后针对不同模态的重要令牌学习模态共享的粗粒度时间依赖性的高效跨模态交互。此外，设计了一个轻量级的门控隐藏投影来控制TIAA的特征流。所得到的模型名为经济型提示语音融合Transformer（EcoCued），与现有的基于Transformer的融合方法和ACSR融合方法相比，在所有现有CS数据集上均达到了最先进的性能。
英文摘要：
Cued Speech (CS) is a pure visual coding method used by hearing-impaired people that combines lip reading with several specific hand shapes to make the spoken language visible. Automatic CS recognition (ACSR) seeks to transcribe visual cues of speech into text, which can help hearing-impaired people to communicate effectively. The visual information of CS contains lip reading and hand cueing, thus the fusion of them plays an important role in ACSR. However, most previous fusion methods struggle to capture the global dependency present in long sequence inputs of multi-modal CS data. As a result, these methods generally fail to learn the effective cross-modal relationships that contribute to the fusion. Recently, attention-based transformers have been a prevalent idea for capturing the global dependency over the long sequence in multi-modal fusion, but existing multi-modal fusion transformers suffer from both poor recognition accuracy and inefficient computation for the ACSR task. To address these problems, we develop a novel computation and parameter efficient multi-modal fusion transformer by proposing a novel Token-Importance-Aware Attention mechanism (TIAA), where a token utilization rate (TUR) is formulated to select the important tokens from the multi-modal streams. More precisely, TIAA firstly models the modality-specific fine-grained temporal dependencies over all tokens of each modality, and then learns the efficient cross-modal interaction for the modality-shared coarse-grained temporal dependencies over the important tokens of different modalities. Besides, a light-weight gated hidden projection is designed to control the feature flows of TIAA. The resulting model, named Economical Cued Speech Fusion Transformer (EcoCued), achieves state-of-the-art performance on all existing CS datasets, compared with existing transformer-based fusion methods and ACSR fusion methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
