基于迭代视觉提示的开放词汇动作定位
Open-Vocabulary Action Localization with Iterative Visual Prompting
AI总结：
本文提出一种免训练的开放词汇视频动作定位方法，通过迭代视觉提示让视觉-语言模型在拼接帧图像上逐步缩小采样窗口，实现与最先进零样本方法相当的定位性能。
摘要：
视频动作定位旨在从长视频中找出特定动作的时间位置。尽管现有的基于学习的方法已经取得了成功，但它们需要对视频进行标注，这会带来相当大的人工成本。本文提出了一种免训练、开放词汇的方法，该方法基于现成的视觉-语言模型（VLMs）。挑战在于，VLMs 既不是为处理长视频而设计的，也不是为查找动作而定制的。我们通过扩展一种迭代视觉提示技术来克服这些问题。具体来说，我们采样视频帧并创建带有帧索引标签的拼接图像，使 VLM 能够识别最有可能对应于动作开始和结束的帧。通过在选定帧周围迭代缩小采样窗口，估计逐渐收敛到更精确的时间边界。我们证明该技术能产生合理的性能，取得与最先进的零样本动作定位相当的结果。这些结果支持将 VLMs 作为理解视频的实用工具。示例代码可在 https://microsoft.github.io/VLM-Video-Action-Localization/ 获取。
英文摘要：
Video action localization aims to find the timings of specific actions from a long video. Although existing learning-based approaches have been successful, they require annotating videos, which comes with a considerable labor cost. This paper proposes a training-free, open-vocabulary approach based on emerging off-the-shelf vision-language models (VLMs). The challenge stems from the fact that VLMs are neither designed to process long videos nor tailored for finding actions. We overcome these problems by extending an iterative visual prompting technique. Specifically, we sample video frames and create a concatenated image with frame index labels, allowing a VLM to identify the frames that most likely correspond to the start and end of the action. By iteratively narrowing the sampling window around the selected frames, the estimation gradually converges to more precise temporal boundaries. We demonstrate that this technique yields reasonable performance, achieving results comparable to state-of-the-art zero-shot action localization. These results support the use of VLMs as a practical tool for understanding videos. Sample code is available at https://microsoft.github.io/VLM-Video-Action-Localization/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
