Global2Local：用于视频描述的联合层次注意力机制
Global2Local: A Joint-Hierarchical Attention for Video Captioning
AI总结：
针对视频描述中捕捉关键语义项及时空相关性的挑战，提出联合层次注意力模型，通过全局到局部的特征选择提升性能，在两个基准数据集上优于现有方法。
摘要：
近年来，自动视频描述受到越来越多的关注，其核心挑战在于从冗余的帧和语义内容中捕捉关键语义项（如对象和动作）及其时空相关性。为此，现有工作要么在全局层面（跨多帧）选择关键视频片段，要么在每个帧内选择关键区域，但忽略了先关键帧后关键区域的层次顺序。本文提出一种新颖的联合层次注意力模型用于视频描述，该模型以层次方式将关键片段、关键帧和关键区域联合嵌入到描述模型中。这种联合层次注意力模型首先进行全局选择以识别关键帧，然后通过Gumbel采样操作基于关键帧进一步识别关键区域，实现准确的全局到局部特征表示以指导描述生成。在MSVD和MSR-VTT两个公开基准数据集上的大量定量评估表明，所提出的方法优于现有最先进方法。
英文摘要：
Recently, automatic video captioning has attracted increasing attention, where the core challenge lies in capturing the key semantic items, like objects and actions as well as their spatial-temporal correlations from the redundant frames and semantic content. To this end, existing works select either the key video clips in a global level~(across multi frames), or key regions within each frame, which, however, neglect the hierarchical order, i.e., key frames first and key regions latter. In this paper, we propose a novel joint-hierarchical attention model for video captioning, which embeds the key clips, the key frames and the key regions jointly into the captioning model in a hierarchical manner. Such a joint-hierarchical attention model first conducts a global selection to identify key frames, followed by a Gumbel sampling operation to identify further key regions based on the key frames, achieving an accurate global-to-local feature representation to guide the captioning. Extensive quantitative evaluations on two public benchmark datasets MSVD and MSR-VTT demonstrates the superiority of the proposed method over the state-of-the-art methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
