arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2211.15076cs.CVcs.AIcs.CL
收藏

面向频率扩散的视频描述精细语义增强

Refined Semantic Enhancement towards Frequency Diffusion for Video Captioning

Wuhan University of Technology(武汉理工大学);Hubei University of Education(湖北第二师范学院);Wuhan University(武汉大学);University of Central Florida(中佛罗里达大学)

Xian Zhong, Zipeng Li, Shuqin Chen, Kui Jiang, Chen Chen, Mang Ye

摘要:

视频描述旨在生成准确描述给定视频的自然语言句子。现有的方法通过在编码阶段探索更丰富的视觉表示或提高解码能力来获得有利的生成。然而,长尾问题阻碍了低频令牌的这些尝试,低频令牌很少出现,但具有关键的语义,在详细生成中起着至关重要的作用。在本文中,我们介绍了一种新的细化语义增强方法频率扩散(RSFD),字幕模型,不断感知的语言表示的不频繁的令牌。具体而言,提出了一个频率感知扩散(FAD)模块来理解低频令牌的语义,以突破生成限制。以这种方式,通过促进吸收出现不足的标记来细化字幕。在FAD的基础上,设计了一个发散语义监督器(Divergent Semantic Supervisor,DSS)模块,以弥补扩散过程中高频词的信息损失,并进一步强调低频词的语义,以缓解长尾问题。大量的实验表明,RSFD在两个基准数据集上的性能优于最先进的方法,即,MSR-VTT和MSVD,表明低频标记语义的增强可以获得竞争生成效果。代码可在https://github.com/lzp870/RSFD上获得。

英文摘要:

Video captioning aims to generate natural language sentences that describe the given video accurately. Existing methods obtain favorable generation by exploring richer visual representations in encode phase or improving the decoding ability. However, the long-tailed problem hinders these attempts at low-frequency tokens, which rarely occur but carry critical semantics, playing a vital role in the detailed generation. In this paper, we introduce a novel Refined Semantic enhancement method towards Frequency Diffusion (RSFD), a captioning model that constantly perceives the linguistic representation of the infrequent tokens. Concretely, a Frequency-Aware Diffusion (FAD) module is proposed to comprehend the semantics of low-frequency tokens to break through generation limitations. In this way, the caption is refined by promoting the absorption of tokens with insufficient occurrence. Based on FAD, we design a Divergent Semantic Supervisor (DSS) module to compensate for the information loss of high-frequency tokens brought by the diffusion process, where the semantics of low-frequency tokens is further emphasized to alleviate the long-tailed problem. Extensive experiments indicate that RSFD outperforms the state-of-the-art methods on two benchmark datasets, i.e., MSR-VTT and MSVD, demonstrate that the enhancement of low-frequency tokens semantics can obtain a competitive generation effect. Code is available at https://github.com/lzp870/RSFD.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑