利用无监督语义信息进行密集视频描述
Dense Video Captioning Using Unsupervised Semantic Information
AI总结：
本文提出无监督语义视觉表示，通过聚类构建视觉码本并编码条目共现概率，在仅用视觉特征的密集视频描述中达到最先进性能。
摘要：
我们提出了一种学习无监督语义视觉信息的方法，其前提是复杂事件可以被分解为更简单的事件，并且这些简单事件在若干复杂事件之间共享。我们首先采用聚类方法对表示进行分组，生成一个视觉码本。随后，我们通过对码本条目的共现概率矩阵进行编码，学习一种密集表示。该表示在仅使用视觉特征的场景中提升了密集视频描述任务的性能。例如，我们在 BMT 方法中替换音频信号，并生成性能相当的时序提案。此外，我们在一个普通 transformer 方法中将视觉表示与我们的描述符拼接，从而在描述子任务上取得相较仅探索视觉特征的方法的最先进性能，同时获得与多模态方法相比具有竞争力的性能。我们的代码可在 https://github.com/valterlej/dvcusi 获取。
英文摘要：
We introduce a method to learn unsupervised semantic visual information based on the premise that complex events can be decomposed into simpler events and that these simple events are shared across several complex events. We first employ a clustering method to group representations producing a visual codebook. Then, we learn a dense representation by encoding the co-occurrence probability matrix for the codebook entries. This representation leverages the performance of the dense video captioning task in a scenario with only visual features. For example, we replace the audio signal in the BMT method and produce temporal proposals with comparable performance. Furthermore, we concatenate the visual representation with our descriptor in a vanilla transformer method to achieve state-of-the-art performance in the captioning subtask compared to the methods that explore only visual features, as well as a competitive performance with multi-modal methods. Our code is available at https://github.com/valterlej/dvcusi.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
