空间和时间的稀疏:与可训练的选择器的视听同步
Sparse in Space and Time: Audio-visual Synchronisation with Trainable Selectors
摘要:
本文的目的是视听同步的一般视频“在野外”。对于这样的视频,可以被利用以用于同步提示的事件在空间上可以是小的,并且可以在许多秒长的视频剪辑期间仅偶尔发生,即同步信号在空间和时间上是“稀疏的”。这与同步说话者视频的情况形成鲜明对比,后者的视听对应在时间和空间上都很密集。我们有四个贡献:(i)为了处理稀疏同步信号所需的较长时间序列,我们设计了一种多模态Transformer模型,该模型采用“选择器”将长音频和视频流转换成小序列,然后将其用于预测流之间的时间偏移。(ii)我们识别了可能来自用于音频和视频的压缩编解码器的伪影,并且可以在训练中由视听模型用于人工解决同步任务。(iii)我们策划一个数据集,只有稀疏的时间和空间同步信号;和(iv)所提出的模型的有效性上密集和稀疏的数据集定量和定性。项目页面:v-iashin.github.io/SparseSync
英文摘要:
The objective of this paper is audio-visual synchronisation of general videos 'in the wild'. For such videos, the events that may be harnessed for synchronisation cues may be spatially small and may occur only infrequently during a many seconds-long video clip, i.e. the synchronisation signal is 'sparse in space and time'. This contrasts with the case of synchronising videos of talking heads, where audio-visual correspondence is dense in both time and space. We make four contributions: (i) in order to handle longer temporal sequences required for sparse synchronisation signals, we design a multi-modal transformer model that employs 'selectors' to distil the long audio and visual streams into small sequences that are then used to predict the temporal offset between streams. (ii) We identify artefacts that can arise from the compression codecs used for audio and video and can be used by audio-visual models in training to artificially solve the synchronisation task. (iii) We curate a dataset with only sparse in time and space synchronisation signals; and (iv) the effectiveness of the proposed model is shown on both dense and sparse datasets quantitatively and qualitatively. Project page: v-iashin.github.io/SparseSync
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。