arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2302.14757cs.MMcs.IRcs.SDeess.AS
收藏

多模态设计文档的音频检索:一种新的数据集和算法

Audio Retrieval for Multimodal Design Documents: A New Dataset and Algorithms

Adobe Research(奥多比研究院)

Prachi Singh, Srikrishna Karanam, Sumit Shekhar

摘要:

我们考虑并提出了一个新的问题,检索音频文件相关的多模态设计文档输入,包括文本元素和视觉图像,例如,生日/贺卡除了增强用户体验之外,集成与这些输入的主题/风格相匹配的音频还有助于提高这些文档的可访问性(例如,视力受损的人可以听音频代替)。虽然最近在音频检索方面的工作存在,但这些方法和数据集明确针对自然图像。然而,我们的问题认为多模态设计文档(由用户使用创意软件创建)与自然点击的照片有很大不同。为此,我们的第一个贡献是收集和策划一个名为Melodic-Design(或MELON)的新的大规模数据集,包括代表各种风格,主题,模板,插图等的设计文档,与音乐音频配对。鉴于我们配对的图像-文本-音频数据集,我们的下一个贡献是一种新的多模态交叉注意音频检索(MMCAR)算法,该算法能够训练神经网络学习跨图像,文本和音频维度的公共共享特征空间。我们使用这些学习到的特征来证明我们的方法优于现有的最先进的方法,并为我们的新数据集上的研究社区提供了一个新的参考基准。

英文摘要:

We consider and propose a new problem of retrieving audio files relevant to multimodal design document inputs comprising both textual elements and visual imagery, e.g., birthday/greeting cards. In addition to enhancing user experience, integrating audio that matches the theme/style of these inputs also helps improve the accessibility of these documents (e.g., visually impaired people can listen to the audio instead). While recent work in audio retrieval exists, these methods and datasets are targeted explicitly towards natural images. However, our problem considers multimodal design documents (created by users using creative software) substantially different from a naturally clicked photograph. To this end, our first contribution is collecting and curating a new large-scale dataset called Melodic-Design (or MELON), comprising design documents representing various styles, themes, templates, illustrations, etc., paired with music audio. Given our paired image-text-audio dataset, our next contribution is a novel multimodal cross-attention audio retrieval (MMCAR) algorithm that enables training neural networks to learn a common shared feature space across image, text, and audio dimensions. We use these learned features to demonstrate that our method outperforms existing state-of-the-art methods and produce a new reference benchmark for the research community on our new dataset.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑