arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2206.06346cs.CV
收藏

通过对象标记的帧-剪辑一致性将图像场景结构引入视频

Bringing Image Scene Structure to Video via Frame-Clip Consistency of Object Tokens

Tel Aviv University(特拉维夫大学);IBM Research(IBM研究院);UC Berkeley(加州大学伯克利分校);MIT-IBM Lab(麻省理工学院-IBM实验室);Google Research(谷歌研究院)

Elad Ben-Avraham, Roei Herzig, Karttikeya Mangalam, Amir Bar, Anna Rohrbach, Leonid Karlinsky, Trevor Darrell, Amir Globerson

摘要:

最近的动作识别模型通过整合对象、它们的位置和交互,取得了令人印象深刻的结果。然而,获得每个帧的密集结构化注释是繁琐且耗时的,使得这些方法训练昂贵且可扩展性较低。与此同时,如果有一小部分带注释的图像可用,无论是在感兴趣的领域之内还是之外,我们如何利用这些图像来完成视频下游任务?我们提出了一个学习框架StructureViT(简称SViT),它演示了如何利用仅在训练期间可用的少量图像的结构来改进视频模型。SViT依赖于两个关键的见解。首先,由于图像和视频都包含结构化信息,因此我们使用一组可跨图像和视频使用的\emph{object tokens}来丰富Transformer模型。第二,视频中各个帧的场景表示应该与静止图像的场景表示"对齐"。这是通过\emph{Frame-Clip Consistency}丢失来实现的,它确保了图像和视频之间的结构化信息流。我们探讨了一个特定的实例化的场景结构,即一个\emph{Hand-Object Graph},包括手和对象与他们的位置作为节点,和物理关系的接触/无接触的边缘。SViT在多个视频理解任务和数据集上显示出强大的性能改进。此外,它还赢得了Ego4D CVPR '22 Object State Localization挑战赛。有关代码和预训练模型,请访问项目页面\url{https://eladb3.github.io/SViT/}

英文摘要:

Recent action recognition models have achieved impressive results by integrating objects, their locations and interactions. However, obtaining dense structured annotations for each frame is tedious and time-consuming, making these methods expensive to train and less scalable. At the same time, if a small set of annotated images is available, either within or outside the domain of interest, how could we leverage these for a video downstream task? We propose a learning framework StructureViT (SViT for short), which demonstrates how utilizing the structure of a small number of images only available during training can improve a video model. SViT relies on two key insights. First, as both images and videos contain structured information, we enrich a transformer model with a set of \emph{object tokens} that can be used across images and videos. Second, the scene representations of individual frames in video should "align" with those of still images. This is achieved via a \emph{Frame-Clip Consistency} loss, which ensures the flow of structured information between images and videos. We explore a particular instantiation of scene structure, namely a \emph{Hand-Object Graph}, consisting of hands and objects with their locations as nodes, and physical relations of contact/no-contact as edges. SViT shows strong performance improvements on multiple video understanding tasks and datasets. Furthermore, it won in the Ego4D CVPR'22 Object State Localization challenge. For code and pretrained models, visit the project page at \url{https://eladb3.github.io/SViT/}

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑