Developments in weakly supervised and self-supervised models could enable speech technology in low-resource settings where full transcriptions are not available. We consider whether keyword localisation is possible using two forms of weak supervision where location information is not provided explicitly. In the first, only the presence or absence of a word is indicated, i.e. a bag-of-words (BoW) labelling. In the second, visual context is provided in the form of an image paired with an unlabelled utterance; a model then needs to be trained in a self-supervised fashion using the paired data. For keyword localisation, we adapt a saliency-based method typically used in the vision domain. We compare this to an existing technique that performs localisation as a part of the network architecture. While the saliency-based method is more flexible (it can be applied without architectural restrictions), we identify a critical limitation when using it for keyword localisation. Of the two forms of supervision, the visually trained model performs worse than the BoW-trained model. We show qualitatively that the visually trained model sometimes locate semantically related words, but this is not consistent. While our results show that there is some signal allowing for localisation, it also calls for other localisation methods better matched to these forms of weak supervision.
YFACC: A Yorùbá speech-image dataset for cross-lingual keyword localisation through visual grounding
Stellenbosch University（斯泰伦博斯大学）；University Politehnica of Bucharest（布加勒斯特理工大学）
视觉接地语音（VGS）模型在与未标记的口头字幕配对的图像上进行训练。这种模型可以用于在不可能获得标记数据的环境中构建语音系统，例如用于记录非书面语言。然而，大多数VGS研究都是用英语或其他高资源语言进行的。本文试图弥补这一缺陷。我们收集并发布了一个新的单扬声器数据集的音频字幕为6 k Flickr图像在Yorkebá-一个真正的低资源语言讲尼日利亚。我们训练一个基于注意力的VGS模型，其中图像自动标记有英语视觉标签，并与Yorkebá话语配对。这使得跨语言的关键字本地化：一个书面的英语查询被检测到，并在Yorkebá语音定位。为了量化较小数据集的影响，我们将其与在类似和更多数据上训练的英语系统进行了比较。我们希望这个新的数据集将刺激对真正的低资源语言使用VGS模型的研究。
Improved Visually Prompted Keyword Localisation in Real Low-Resource Settings
Electrical and Electronic Engineering（电气与电子工程）；Stellenbosch University（斯泰伦博斯大学）；National University of Science and Technology（国家科学与技术大学）；\textsc{Politehnica} Bucharest（布加勒斯特理工大学）
