RealTranS：基于卷积加权收缩Transformer的端到端同步语音翻译
RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformer
摘要：
端到端同步语音翻译（SST），直接将一种语言的语音实时翻译成另一种语言的文本，在许多情况下都很有用，但尚未得到充分的研究。在这项工作中，我们提出了RealTransS，SST的端到端模型。为了弥合语音和文本之间的模态差距，RealTransS使用交错卷积和单向Transformer层逐渐对输入语音进行下采样，以进行声学建模，然后使用加权收缩操作和语义编码器将语音特征映射到文本空间。此外，为了提高模型在同时场景中的性能，我们提出了一个空白惩罚，以提高收缩质量和等待K步N策略，允许在解码过程中的局部重新排序。在公共数据集和广泛使用的数据集上的实验表明，具有Wait-K-Stride-N策略的RealTransS在不同延迟设置下的性能优于先前的端到端模型以及级联模型。
英文摘要：
End-to-end simultaneous speech translation (SST), which directly translates speech in one language into text in another language in real-time, is useful in many scenarios but has not been fully investigated. In this work, we propose RealTranS, an end-to-end model for SST. To bridge the modality gap between speech and text, RealTranS gradually downsamples the input speech with interleaved convolution and unidirectional Transformer layers for acoustic modeling, and then maps speech features into text space with a weighted-shrinking operation and a semantic encoder. Besides, to improve the model performance in simultaneous scenarios, we propose a blank penalty to enhance the shrinking quality and a Wait-K-Stride-N strategy to allow local reranking during decoding. Experiments on public and widely-used datasets show that RealTranS with the Wait-K-Stride-N strategy outperforms prior end-to-end models as well as cascaded models in diverse latency settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
