End-to-end Speech Translation (E2E ST) aims to directly translate source speech into target text. Existing ST methods perform poorly when only extremely small speech-text data are available for training. We observe that an ST model's performance closely correlates with its embedding similarity between speech and source transcript. In this paper, we propose Word-Aligned COntrastive learning (WACO), a simple and effective method for extremely low-resource speech-to-text translation. Our key idea is bridging word-level representations for both speech and text modalities via contrastive learning. We evaluate WACO and other methods on the MuST-C dataset, a widely used ST benchmark, and on a low-resource direction Maltese-English from IWSLT 2023. Our experiments demonstrate that WACO outperforms the best baseline by 9+ BLEU points with only 1-hour parallel ST data. Code is available at https://github.com/owaski/WACO.
BilBOWA: Fast Bilingual Distributed Representations without Word Alignments
Google Inc.（谷歌公司）；Université de Montréal（蒙特利尔大学）；Canadian Institute for Advanced Research（加拿大高等研究院）
我们介绍了BilBOWA（Bilingual Bag-of-Words without Alignments），这是一种简单且计算效率高的模型，用于学习单词的双语分布式表示，可以扩展到大型单语数据集，并且不需要单词对齐的并行训练数据。相反，它直接在单语数据上训练，并从较小的原始文本对齐数据集中提取双语信号。这是使用一种新的采样词袋跨语言目标来实现的，该目标用于正则化两个噪声对比语言模型，以实现高效的跨语言特征学习。我们表明，使用所提出的模型学习的双语嵌入在跨语言文档分类任务以及WMT 11数据的词汇翻译任务上优于最先进的方法。
