arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.01616cs.CLcs.IRcs.SDeess.AS
收藏

将大语言模型(LLM)转化为跨模态跨语言检索系统

Transforming LLMs into Cross-modal and Cross-lingual Retrieval Systems

Boston University(波士顿大学);The University of Edinburgh(爱丁堡大学);Google Research(谷歌研究院);Google DeepMind(谷歌DeepMind)

Frank Palma Gomez, Ramon Sanabria, Yun-hsuan Sung, Daniel Cer, Siddharth Dalmia, Gustavo Hernandez Abrego

AI总结:

本研究提出用大语言模型初始化多模态双编码器检索系统,仅用21种语言训练即可实现102种语言的语音文本匹配,平均Recall@1提升10%,还可借助机器翻译数据增强跨语言匹配能力。

摘要:

大语言模型(LLM)基于纯文本数据训练,训练数据覆盖的语言远不止有语音-文本配对数据的语言。与此同时,基于双编码器(Dual Encoder, DE)的检索系统将查询和文档映射到同一嵌入空间,已在检索和双语文本挖掘中展现出成效。为实现多语言下的语音与文本匹配,我们提出利用LLM初始化多模态双编码器检索系统。与传统方法不同,我们的系统在LLM预训练阶段无需语音数据,且能借助LLM的多语言文本理解能力,匹配检索训练阶段未见过的语言的语音与文本。我们的多模态基于LLM的检索系统仅在21种语言上训练,却能实现102种语言的语音-文本匹配,性能优于此前专门在全部102种语言上训练的系统,在这些语言的平均Recall@1指标上实现了10%的绝对提升。此外,我们的模型展现出跨语言语音-文本匹配能力,利用现成的机器翻译数据可进一步增强该能力。

英文摘要:

Large language models (LLMs) are trained on text-only data that go far beyond the languages with paired speech and text data. At the same time, Dual Encoder (DE) based retrieval systems project queries and documents into the same embedding space and have demonstrated their success in retrieval and bi-text mining. To match speech and text in many languages, we propose using LLMs to initialize multi-modal DE retrieval systems. Unlike traditional methods, our system doesn't require speech data during LLM pre-training and can exploit LLM's multilingual text understanding capabilities to match speech and text in languages unseen during retrieval training. Our multi-modal LLM-based retrieval system is capable of matching speech and text in 102 languages despite only training on 21 languages. Our system outperforms previous systems trained explicitly on all 102 languages. We achieve a 10% absolute improvement in Recall@1 averaged across these languages. Additionally, our model demonstrates cross-lingual speech and text matching, which is further enhanced by readily available machine translation data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑