挖掘意义:通过多网络共识从视觉到语言
Mining for meaning: from vision to language through multiple networks consensus
摘要:
将视觉数据描述成自然语言是一项非常具有挑战性的任务,它处于计算机视觉、自然语言处理和机器学习的交叉点。语言远远超出了对物理对象及其相互作用的描述,可以以多种方式传达相同的抽象概念。它既是关于最高语义层次的内容,也是关于流畅的形式。在这里,我们提出了一种方法来描述视频在自然语言之间达成共识的多个编码器-解码器网络。找到这样一个共识的语言描述,它与一个更大的群体共享共同的属性,有更好的机会来传达正确的含义。我们提出并训练了几种网络架构,并使用不同类型的图像,音频和视频功能。每个模型都会产生自己对输入视频的描述,最好的一个是通过一个高效的两阶段共识过程选择的。我们通过在具有挑战性的MSR-VTT数据集上获得最先进的结果来证明我们方法的优势。
英文摘要:
Describing visual data into natural language is a very challenging task, at the intersection of computer vision, natural language processing and machine learning. Language goes well beyond the description of physical objects and their interactions and can convey the same abstract idea in many ways. It is both about content at the highest semantic level as well as about fluent form. Here we propose an approach to describe videos in natural language by reaching a consensus among multiple encoder-decoder networks. Finding such a consensual linguistic description, which shares common properties with a larger group, has a better chance to convey the correct meaning. We propose and train several network architectures and use different types of image, audio and video features. Each model produces its own description of the input video and the best one is chosen through an efficient, two-phase consensus process. We demonstrate the strength of our approach by obtaining state of the art results on the challenging MSR-VTT dataset.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。