多模态神经数据库
Multimodal Neural Databases
AI总结:
针对现有多媒体信息检索方法无法支持类数据库查询的问题,提出多模态神经数据库框架MMNDBs,可跨模态推理回答复杂查询,验证了现有模型局限并展现了技术潜力。
摘要:
通过文本、图像及其他模态可获取的松散结构化数据不断增多,亟需新的查询方式。多媒体信息检索填补了这一空白,且近年来取得了令人瞩目的进展。在多模态深度学习最新进展的极大推动下,大型多媒体档案的搜索与检索等任务的性能得到了大幅提升。然而,该领域的方法所支持的查询类型仍然有限,尤其无法回答类数据库查询。为此,受神经数据库相关最新研究的启发,我们提出了一个名为Multimodal Neural Databases(多模态神经数据库,简称MMNDBs)的新框架。MMNDBs能够大规模回答涉及对文本、图像等不同输入模态进行推理的复杂类数据库查询。本文提出了首个能够满足这一系列要求的架构,并通过多个基准对其进行测试,揭示了现有模型的局限性。结果表明,这些新技术具备处理来自不同模态的非结构化数据的潜力,为该领域的未来研究铺平了道路。复现实验的代码将发布于https://github.com/GiovanniTRA/MultimodalNeuralDatabases。
英文摘要:
The rise in loosely-structured data available through text, images, and other modalities has called for new ways of querying them. Multimedia Information Retrieval has filled this gap and has witnessed exciting progress in recent years. Tasks such as search and retrieval of extensive multimedia archives have undergone massive performance improvements, driven to a large extent by recent developments in multimodal deep learning. However, methods in this field remain limited in the kinds of queries they support and, in particular, their inability to answer database-like queries. For this reason, inspired by recent work on neural databases, we propose a new framework, which we name Multimodal Neural Databases (MMNDBs). MMNDBs can answer complex database-like queries that involve reasoning over different input modalities, such as text and images, at scale. In this paper, we present the first architecture able to fulfill this set of requirements and test it with several baselines, showing the limitations of currently available models. The results show the potential of these new techniques to process unstructured data coming from different modalities, paving the way for future research in the area. Code to replicate the experiments will be released at https://github.com/GiovanniTRA/MultimodalNeuralDatabases
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。