arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2304.10316cs.CV
收藏

Search-Map-Search:一种用于动作识别的帧选择范式

Search-Map-Search: A Frame Selection Paradigm for Action Recognition

University of Alberta(阿尔伯塔大学);Tencent(腾讯)

Mingjun Zhao, Yakun Yu, Xiaoli Wang, Lei Yang, Di Niu

AI总结:

本文提出 Search-Map-Search 帧选择范式,通过分层搜索、特征映射与推理期搜索建模帧间交互,以低开销提升动作识别性能。

摘要:

尽管深度学习在视频理解任务中取得了成功,但处理视频中的每一帧在计算上代价高昂,而且在实时应用中往往没有必要。帧选择旨在提取最具信息量和代表性的帧,以帮助模型更好地理解视频内容。现有的帧选择方法要么基于逐帧重要性预测单独采样帧,而不考虑帧之间的交互;要么采用强化学习智能体连续寻找代表性帧,这类方法训练成本高昂,并可能导致潜在的稳定性问题。为克服现有方法的局限,我们提出了一种 Search-Map-Search 学习范式,它结合了启发式搜索和监督学习的优势,将视频中最佳帧组合作为一个整体进行选择。通过将搜索与学习相结合,所提出的方法能够更好地捕捉帧间交互,同时产生较低的推理开销。具体而言,我们首先提出一种在每个训练视频上进行的分层搜索方法,以搜索在下游任务上误差最低的最优帧组合。随后学习一个特征映射函数,将视频的帧映射到其目标最优帧组合的表示。在推理期间,对未见过的视频执行另一次搜索,以选择其特征表示接近投影特征表示的帧组合。基于多个动作识别基准的大量实验表明,我们的帧选择方法有效提升了动作识别模型的性能,并显著优于若干具有竞争力的基线。

英文摘要:

Despite the success of deep learning in video understanding tasks, processing every frame in a video is computationally expensive and often unnecessary in real-time applications. Frame selection aims to extract the most informative and representative frames to help a model better understand video content. Existing frame selection methods either individually sample frames based on per-frame importance prediction, without considering interaction among frames, or adopt reinforcement learning agents to find representative frames in succession, which are costly to train and may lead to potential stability issues. To overcome the limitations of existing methods, we propose a Search-Map-Search learning paradigm which combines the advantages of heuristic search and supervised learning to select the best combination of frames from a video as one entity. By combining search with learning, the proposed method can better capture frame interactions while incurring a low inference overhead. Specifically, we first propose a hierarchical search method conducted on each training video to search for the optimal combination of frames with the lowest error on the downstream task. A feature mapping function is then learned to map the frames of a video to the representation of its target optimal frame combination. During inference, another search is performed on an unseen video to select a combination of frames whose feature representation is close to the projected feature representation. Extensive experiments based on several action recognition benchmarks demonstrate that our frame selection method effectively improves performance of action recognition models, and significantly outperforms a number of competitive baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑