arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2308.09685cs.LGcs.CVcs.MMcs.SDeess.AS
收藏

时间中的视听瞬间:一个大规模带标注的视听动作数据集

Audiovisual Moments in Time: A Large-Scale Annotated Dataset of Audiovisual Actions

University of Birmingham(伯明翰大学);University of Haifa(海法大学);Donders Institute for Brain, Cognition and Behaviour(唐德斯脑、认知与行为研究所);Radboud University(拉德堡德大学)

Michael Joannou, Pia Rotshtein, Uta Noppeney

AI总结:

本文构建大规模视听动作数据集AVMIT,提供人工标注、测试集与预计算视听特征嵌入,并证明仅用视听事件训练可显著提升RNN识别准确率。

摘要:

我们提出了Audiovisual Moments in Time(AVMIT),这是一个大规模的视听动作事件数据集。在一项大规模标注任务中,11名参与者对来自Moments in Time数据集(MIT)的3秒视听视频子集进行了标注。对于每次试验,参与者评估被标注的视听动作事件是否存在,以及它是否是视频中最突出的特征。该数据集包含57,177个视听视频的标注,每个视频均由11名受过训练的参与者中的3名独立评估。基于这一初始收集,我们创建了一个精选测试集,涵盖16个不同的动作类别,每类60个视频(共960个视频)。我们还提供了两组预计算的视听特征嵌入,音频数据使用VGGish/YamNet,视觉数据使用VGG16/EfficientNetB0,从而降低视听DNN研究的入门门槛。我们探索了AVMIT标注和特征嵌入在提升视听事件识别性能方面的优势。一系列6个循环神经网络(RNN)分别在AVMIT过滤后的视听事件或来自MIT的模态无关事件上进行训练,然后在我们的视听测试集上测试。在所有RNN中,仅通过在视听事件上训练,top 1准确率提高了2.71-5.94%,其效果甚至超过训练数据增加三倍。我们预计,新标注的AVMIT数据集将成为涉及计算模型和人类参与者的研究与对比实验的宝贵资源,尤其是在解决视听对应关系至关重要的研究问题时。

英文摘要:

We present Audiovisual Moments in Time (AVMIT), a large-scale dataset of audiovisual action events. In an extensive annotation task 11 participants labelled a subset of 3-second audiovisual videos from the Moments in Time dataset (MIT). For each trial, participants assessed whether the labelled audiovisual action event was present and whether it was the most prominent feature of the video. The dataset includes the annotation of 57,177 audiovisual videos, each independently evaluated by 3 of 11 trained participants. From this initial collection, we created a curated test set of 16 distinct action classes, with 60 videos each (960 videos). We also offer 2 sets of pre-computed audiovisual feature embeddings, using VGGish/YamNet for audio data and VGG16/EfficientNetB0 for visual data, thereby lowering the barrier to entry for audiovisual DNN research. We explored the advantages of AVMIT annotations and feature embeddings to improve performance on audiovisual event recognition. A series of 6 Recurrent Neural Networks (RNNs) were trained on either AVMIT-filtered audiovisual events or modality-agnostic events from MIT, and then tested on our audiovisual test set. In all RNNs, top 1 accuracy was increased by 2.71-5.94\% by training exclusively on audiovisual events, even outweighing a three-fold increase in training data. We anticipate that the newly annotated AVMIT dataset will serve as a valuable resource for research and comparative experiments involving computational models and human participants, specifically when addressing research questions where audiovisual correspondence is of critical importance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑