Talk，Don 't Write：一种基于直接语音的图像检索方法研究
Talk, Don't Write: A Study of Direct Speech-Based Image Retrieval
摘要：
基于语音的图像检索已经被研究作为联合表示学习的代理，通常不强调检索本身。因此，目前还不清楚基于语音的检索在实践中的效果如何--无论是在绝对意义上，还是与将自动语音识别（ASR）与强大的文本编码器相结合的替代策略相比。在这项工作中，我们广泛研究和扩展编码器架构，训练方法（包括单峰和多峰预训练）和其他因素的选择。我们的实验涵盖了三个数据集的不同类型的语音：Flickr音频，地点音频和本地化叙事。我们最好的模型配置实现了比现有技术更大的增益，例如，Flickr Audio的一次召回率从21.8%提高到33.2%，Places Audio的一次召回率从27.6%提高到53.4%。我们还展示了我们最好的基于语音的模型可以匹配或超过级联ASR到文本编码时，语音是自发的，重音，或以其他方式难以自动转录。
英文摘要：
Speech-based image retrieval has been studied as a proxy for joint representation learning, usually without emphasis on retrieval itself. As such, it is unclear how well speech-based retrieval can work in practice -- both in an absolute sense and versus alternative strategies that combine automatic speech recognition (ASR) with strong text encoders. In this work, we extensively study and expand choices of encoder architectures, training methodology (including unimodal and multimodal pretraining), and other factors. Our experiments cover different types of speech in three datasets: Flickr Audio, Places Audio, and Localized Narratives. Our best model configuration achieves large gains over state of the art, e.g., pushing recall-at-one from 21.8% to 33.2% for Flickr Audio and 27.6% to 53.4% for Places Audio. We also show our best speech-based models can match or exceed cascaded ASR-to-text encoding when speech is spontaneous, accented, or otherwise hard to automatically transcribe.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
