长音频访谈与问题的时序对齐：多模态数据集成案例研究
Temporally Aligning Long Audio Interviews with Questions: A Case Study in Multimodal Data Integration
AI总结：
本研究提出INDENT框架，利用交叉注意力模型和句子时序先验，在长音频健康调查中定位问题对应片段，在11种印度语言上验证了有效性，R-avg提升约3%。
摘要：
音频与文本对齐问题在训练期间使用完全监督的方法已得到大量研究。然而，这通常不适用于长音频记录的场景，其中被查询的文本不会在音频文件中逐字出现。本研究是与一个名为CARE India的非政府组织合作开展的，该组织收集印度比哈尔邦农村地区年轻母亲的长时间音频健康调查。给定一份用于指导这些调查的问卷中的问题，我们的目标是定位该问题在长音频记录中被提出的位置。这对非洲和亚洲的组织非常有价值，否则他们不得不费力地浏览长且嘈杂的音频记录来定位感兴趣的问题（和答案）。我们提出的框架INDENT使用基于交叉注意力的模型以及关于句子时间顺序的先验信息来学习语音嵌入，这些嵌入捕获了底层口语文本的语义。这些学习到的嵌入用于在推理时基于文本查询检索相应的音频片段。我们通过实验证明了我们的模型相对于基于文本启发式方法所获得的结果具有显著的有效性（R-avg改进约3%）。我们还展示了使用最先进的印度语言ASR模型生成的嘈杂ASR在替代语音时能产生更好的结果。仅使用印地语数据训练的INDENT能够服务于（语义上）共享文本空间所支持的所有语言。我们在11种印度语言上对此进行了实证说明。
英文摘要：
The problem of audio-to-text alignment has seen significant amount of research using complete supervision during training. However, this is typically not in the context of long audio recordings wherein the text being queried does not appear verbatim within the audio file. This work is a collaboration with a non-governmental organization called CARE India that collects long audio health surveys from young mothers residing in rural parts of Bihar, India. Given a question drawn from a questionnaire that is used to guide these surveys, we aim to locate where the question is asked within a long audio recording. This is of great value to African and Asian organizations that would otherwise have to painstakingly go through long and noisy audio recordings to locate questions (and answers) of interest. Our proposed framework, INDENT, uses a cross-attention-based model and prior information on the temporal ordering of sentences to learn speech embeddings that capture the semantics of the underlying spoken text. These learnt embeddings are used to retrieve the corresponding audio segment based on text queries at inference time. We empirically demonstrate the significant effectiveness (improvement in R-avg of about 3%) of our model over those obtained using text-based heuristics. We also show how noisy ASR, generated using state-of-the-art ASR models for Indian languages, yields better results when used in place of speech. INDENT, trained only on Hindi data is able to cater to all languages supported by the (semantically) shared text space. We illustrate this empirically on 11 Indic languages.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
