首页/论文/2302.00765

arXiv 2302.007652023-02-01cs.CLcs.SDeess.AS

Visually Grounded Keyword Detection and Localisation for Low-Resource Languages

Stellenbosch University（斯泰伦博斯大学）；Aerobotics（Aerobotics公司）

Kayode Kolawole Olaleye

备注：: PhD dissertation, University of Stellenbosch, 108 pages, submitted and accepted 2023

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

This study investigates the use of Visually Grounded Speech (VGS) models for keyword localisation in speech. The study focusses on two main research questions: (1) Is keyword localisation possible with VGS models and (2) Can keyword localisation be done cross-lingually in a real low-resource setting? Four methods for localisation are proposed and evaluated on an English dataset, with the best-performing method achieving an accuracy of 57%. A new dataset containing spoken captions in Yoruba language is also collected and released for cross-lingual keyword localisation. The cross-lingual model obtains a precision of 16% in actual keyword localisation and this performance can be improved by initialising from a model pretrained on English data. The study presents a detailed analysis of the model's success and failure modes and highlights the challenges of using VGS models for keyword localisation in low-resource settings.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

改进的视觉提示关键词在现实低资源环境中的本地化

Improved Visually Prompted Keyword Localisation in Real Low-Resource Settings

Electrical and Electronic Engineering（电气与电子工程）；Stellenbosch University（斯泰伦博斯大学）；National University of Science and Technology（国家科学与技术大学）；\textsc{Politehnica} Bucharest（布加勒斯特理工大学）

本文提出了一种无需转录的少样本学习方法，用于在低资源语言中实现视觉提示关键词本地化，并在约鲁巴语言上验证了其有效性。

2025-11-26cs.CLcs.CVeess.AS

Towards visually prompted keyword localisation for zero-resource spoken languages

Stellenbosch University（斯泰伦博斯大学）

Imagine being able to show a system a visual depiction of a keyword and finding spoken utterances that contain this keyword from a zero-resource speech corpus. We formalise this task and call it visually prompted keyword localisation (VPKL): given an image of a keyword, detect and predict where in an utterance the keyword occurs. To do VPKL, we propose a speech-vision model with a novel localising attention mechanism which we train with a new keyword sampling scheme. We show that these innovations give improvements in VPKL over an existing speech-vision model. We also compare to a visual bag-o…

2022-10-12cs.CLcs.SDeess.AS

Attention-Based Keyword Localisation in Speech using Visual Grounding

Stellenbosch University（斯泰伦博斯大学）

Visually grounded speech models learn from images paired with spoken captions. By tagging images with soft text labels using a trained visual classifier with a fixed vocabulary, previous work has shown that it is possible to train a model that can detect whether a particular text keyword occurs in speech utterances or not. Here we investigate whether visually grounded speech models can also do keyword localisation: predicting where, within an utterance, a given textual keyword occurs without any explicit text-based or alignment supervision. We specifically consider whether incorporating attent…

2021-06-23cs.CLcs.SDeess.AS

Keyword localisation in untranscribed speech using visually grounded speech models

Keyword localisation is the task of finding where in a speech utterance a given query keyword occurs. We investigate to what extent keyword localisation is possible using a visually grounded speech (VGS) model. VGS models are trained on unlabelled images paired with spoken captions. These models are therefore self-supervised -- trained without any explicit textual label or location information. To obtain training targets, we first tag training images with soft text labels using a pretrained visual classifier with a fixed vocabulary. This enables a VGS model to predict the presence of a written…

2022-02-02cs.CLcs.SDeess.AS

面向低资源语言与认知建模的视觉接地语音模型

Visually Grounded Speech Models for Low-resource Languages and Cognitive Modelling

Stellenbosch University（斯泰伦博斯大学）

本文研究视觉接地语音模型在低资源语言少样本关键词定位中的应用，并分析其互斥性偏置，发现多语性对该偏置的影响与儿童不同。

2024-09-03cs.CLcs.CV

Towards localisation of keywords in speech using weak supervision

Stellenbosch University（斯泰伦博斯大学）

Developments in weakly supervised and self-supervised models could enable speech technology in low-resource settings where full transcriptions are not available. We consider whether keyword localisation is possible using two forms of weak supervision where location information is not provided explicitly. In the first, only the presence or absence of a word is indicated, i.e. a bag-of-words (BoW) labelling. In the second, visual context is provided in the form of an image paired with an unlabelled utterance; a model then needs to be trained in a self-supervised fashion using the paired data. Fo…

2020-12-14cs.CLeess.AS

更多相关论文收起更多论文

YFACC: A Yorùbá speech-image dataset for cross-lingual keyword localisation through visual grounding

Stellenbosch University（斯泰伦博斯大学）；University Politehnica of Bucharest（布加勒斯特理工大学）

Visually grounded speech (VGS) models are trained on images paired with unlabelled spoken captions. Such models could be used to build speech systems in settings where it is impossible to get labelled data, e.g. for documenting unwritten languages. However, most VGS studies are in English or other high-resource languages. This paper attempts to address this shortcoming. We collect and release a new single-speaker dataset of audio captions for 6k Flickr images in Yorùbá -- a real low-resource language spoken in Nigeria. We train an attention-based VGS model where images are automatically tagged…

2022-10-12cs.CLeess.AS

Collection and Evaluation of a Long-Term 4D Agri-Robotic Dataset

Lincoln Centre for Autonomous Systems (LCAS)（林肯自主系统中心（LCAS））；University of Lincoln（林肯大学）

Long-term autonomy is one of the most demanded capabilities looked into a robot. The possibility to perform the same task over and over on a long temporal horizon, offering a high standard of reproducibility and robustness, is appealing. Long-term autonomy can play a crucial role in the adoption of robotics systems for precision agriculture, for example in assisting humans in monitoring and harvesting crops in a large orchard. With this scope in mind, we report an ongoing effort in the long-term deployment of an autonomous mobile robot in a vineyard for data collection across multiple months. …

2022-11-25cs.CVcs.LGcs.RO

语音中基于视觉的跨语言关键词识别

Visually grounded cross-lingual keyword spotting in speech

Stellenbosch University（斯泰伦博斯大学）；Saarland University（萨尔大学）

最近的工作考虑了如何与语音配对的图像可以被用来作为监督时，transparency是不可用的语音系统的建设。我们问视觉接地是否可以用于跨语言的关键字发现：给定一种语言的文本关键字，任务是检索包含另一种语言的关键字的口语。这可以使得能够使用高资源语言中的文本查询来搜索低资源语言中的语音。作为概念验证，我们使用英语语音和德语查询：我们使用德语视觉标记器为每个训练图像添加关键字标签，然后训练神经网络将英语语音映射到德语关键字。在没有并行语音翻译或翻译的情况下，该模型的精度达到58%。我们发现，大多数错误的检索包含等效或语义相关的关键字，排除这些将提高P@10至91%。

2018-06-13cs.CLcs.CV

基于视觉的未转录语音关键词预测学习

Visually grounded learning of keyword prediction from untranscribed speech

Toyota Technological Institute at Chicago（芝加哥丰田技术研究院）

在语言习得过程中，婴儿有视觉线索的好处，以地面口语。机器人同样可以使用音频和视觉传感器。最近的工作表明，图像和口头字幕可以映射到一个有意义的共同空间，允许图像检索使用语音，反之亦然。在这种设置的图像与未转录的口头字幕配对，我们考虑是否可以使用计算机视觉系统来获得语音的文本标签。具体地说，我们使用一个图像到文字的多标签视觉分类器标记图像与软文本标签，然后训练一个神经网络映射到这些软目标的语音。我们表明，由此产生的语音系统是能够预测哪些词出现在一个话语-作为一个口头的词袋分类-没有看到任何平行的语音和文本。我们发现，该模型经常混淆语义相关的词，例如“人”和“人”，使其更有效的语义关键字spotter。

2017-05-25cs.CLcs.CV

通过图像连接语音与文字

Connecting Speech to Words through Images

Ministry of Research, Innovation and Digitization, CNCS-UEFISCDI（研究、创新和数字化部，CNCS-UEFISCDI）

提出一种基于视觉的方法，利用图像和语音描述构建口语词汇表，无需文本监督，在口语词检索和关键词检测中优于神经基线。

2026-06-16cs.CL

基于未转录语音可视化模型的语义语音检索

Semantic speech retrieval with a visually grounded model of untranscribed speech

Stellenbosch University（斯泰伦博斯大学）；TTI-Chicago（芝加哥丰田技术研究所）

人们对可以从与视觉上下文配对的未标记语音中学习的模型越来越感兴趣。此设置与低资源语音处理、机器人和人类语言习得研究相关。在这里，我们研究了一个基于视觉的语音模型，在与口头字幕配对的场景图像上训练，如何捕捉语义的各个方面。我们使用外部图像标记器从图像中生成软文本标签，这些标签作为神经模型的目标，将未转录的语音映射到（语义）关键字标签。我们引入了一个新收集的数据集的人类语义相关性判断和相关的任务，语义语音检索，其目标是搜索口语的语义相关的一个给定的文本查询。在没有看到任何文本的情况下，在并行语音和图像上训练的模型在其十大语义检索中达到了近60%的精度。与在transmittance上训练的监督模型相比，我们的模型在某些方面更好地匹配人类的判断，特别是在检索非逐字语义匹配方面。我们进行了广泛的分析模型及其产生的表示。

2018-10-31cs.CLcs.CVeess.AS

仅使用视觉 grounding 将不同语言的书面词映射到口语词

Mapping Written Words to Spoken Words in a Different Language Using Only Visual Grounding

Politehnica Bucharest（布加勒斯特理工大学）；Stellenbosch University（斯坦陵布什大学）；Trinity College Dublin（都柏林圣三一学院）

本研究针对低资源场景下的语音数据构建问题，提出一种基于自监督语音表示的对齐方法，可从视觉 grounding 数据中直接学习跨语言词到语音的映射，效果优于以往的注意力模型。

2026-08-28cs.CLeess.AS

VLD-RAG：用于长的、视觉丰富的多页文档的智能视觉语言检索增强生成

VLD-RAG: Agentic Vision-Language Retrieval-Augmented Generation for Long, Visually-Rich Multi-Page Documents

Mazelone（马泽隆）

研究针对视觉丰富的长文档问答的多模态检索增强生成，提出VLD-RAG框架，构建多模态索引并采用混合检索策略，经智能体工作流程协调，在相关基准上提升了证据页面检索及问答表现，凸显协调验证与混合检索的重要性。

2026-07-29cs.AIcs.CLcs.IR

xDBTagger: Explainable Natural Language Interface to Databases Using Keyword Mappings and Schema Graph

University of Waterloo（滑铁卢大学）；The Central Bank of the Republic of Türkiye（土耳其共和国中央银行）；Bilkent University（比尔肯特大学）

Translating natural language queries (NLQ) into structured query language (SQL) in interfaces to relational databases is a challenging task that has been widely studied by researchers from both the database and natural language processing communities. Numerous works have been proposed to attack the natural language interfaces to databases (NLIDB) problem either as a conventional pipeline-based or an end-to-end deep-learning-based solution. Nevertheless, regardless of the approach preferred, such solutions exhibit black-box nature, which makes it difficult for potential users targeted by these …

2022-10-07cs.AIcs.CLcs.DB