最近的动作识别模型通过整合对象、它们的位置和交互,取得了令人印象深刻的结果。然而,获得每个帧的密集结构化注释是繁琐且耗时的,使得这些方法训练昂贵且可扩展性较低。与此同时,如果有一小部分带注释的图像可用,无论是在感兴趣的领域之内还是之外,我们如何利用这些图像来完成视频下游任务?我们提出了一个学习框架StructureViT(简称SViT),它演示了如何利用仅在训练期间可用的少量图像的结构来改进视频模型。SViT依赖于两个关键的见解。首先,由于图像和视频都包含结构化信息,因此我们使用一组可跨图像和视频使用的\emph{object tokens}来丰富Transformer模型。第二,视频中各个帧的场景表示应该与静止图像的场景表示"对齐"。这是通过\emph{Frame-Clip Consistency}丢失来实现的,它确保了图像和视频之间的结构化信息流。我们探讨了一个特定的实例化的场景结构,即一个\emph{Hand-Object Graph},包括手和对象与他们的位置作为节点,和物理关系的接触/无接触的边缘。SViT在多个视频理解任务和数据集上显示出强大的性能改进。此外,它还赢得了Ego4D CVPR '22 Object State Localization挑战赛。有关代码和预训练模型,请访问项目页面\url{https://eladb3.github.io/SViT/}
英文摘要:
Recent action recognition models have achieved impressive results by integrating objects, their locations and interactions. However, obtaining dense structured annotations for each frame is tedious and time-consuming, making these methods expensive to train and less scalable. At the same time, if a small set of annotated images is available, either within or outside the domain of interest, how could we leverage these for a video downstream task? We propose a learning framework StructureViT (SViT for short), which demonstrates how utilizing the structure of a small number of images only available during training can improve a video model. SViT relies on two key insights. First, as both images and videos contain structured information, we enrich a transformer model with a set of \emph{object tokens} that can be used across images and videos. Second, the scene representations of individual frames in video should "align" with those of still images. This is achieved via a \emph{Frame-Clip Consistency} loss, which ensures the flow of structured information between images and videos. We explore a particular instantiation of scene structure, namely a \emph{Hand-Object Graph}, consisting of hands and objects with their locations as nodes, and physical relations of contact/no-contact as edges. SViT shows strong performance improvements on multiple video understanding tasks and datasets. Furthermore, it won in the Ego4D CVPR'22 Object State Localization challenge. For code and pretrained models, visit the project page at \url{https://eladb3.github.io/SViT/}
MA-FSAR: Multimodal Adaptation of CLIP for Few-Shot Action Recognition
College of Control Science and Engineering, Zhejiang University(浙江大学控制科学与工程学院);Alibaba Group(阿里巴巴集团);College of Computer Science and Technology, Zhejiang University of Technology(浙江工业大学计算机科学与技术学院);SGIT AI Lab, State Grid Corporation of China(中国国家电网有限公司SGIT人工智能实验室);Baidu(百度);Institute of AI (TeleAI)(中国电信人工智能研究院(TeleAI));School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University(西北工业大学人工智能与光学电子学院(iOPEN))
Bringing CLIP to the Clinic: Dynamic Soft Labels and Negation-Aware Learning for Medical Analysis
Interdisciplinary Program in Bioengineering, Seoul National University Graduate School(生物工程跨学科项目,首尔国立大学研究生院);Integrated Major in Innovative Medical Science, Seoul National University Graduate School(创新医学联合专业,首尔国立大学研究生院);Department of Radiology, Seoul National University Hospital(放射科,首尔国立大学医院)
Parameter-Efficient CLIP Adaptation for 3D Understanding via Unified Tokenization
Fondazione Bruno Kessler(布鲁诺·科塞拉基金会);University of Trento(特伦托大学);University of Pisa(比萨大学);Beijing Forestry University(北京林业大学);Shanghai Jiao Tong University(上海交通大学);Hong Kong University of Science and Technology (GZ)(香港科技大学);Shandong University(山东大学);University of California, Merced(加州大学默塞德分校)
Enhancing Vision-Language Model with Unmasked Token Alignment
Institute for AI Industry Research (AIR)(人工智能产业研究院);Tsinghua University(清华大学);Sensetime Research(商汤科技研究院);CPII under InnoHK(InnoHK下属CPII);Shanghai AI Laboratory(上海人工智能实验室)
Improving CLIP Adaptation by Breaking Tail Alignment for Source-Free Cross-Domain Few-Shot Learning
School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China(华中科技大学计算机科学与技术学院);Institute of Artificial Intelligence, Huazhong University of Science and Technology, Wuhan, China(华中科技大学人工智能研究院)
CLIP-ReID: Exploiting Vision-Language Model for Image Re-Identification without Concrete Text Labels
Shanghai Key Laboratory of Multidimensional Information Processing(上海市多维度信息处理重点实验室);Key Laboratory of Advanced Theory and Application in Statistics and Data Science(统计与数据科学前沿理论及应用重点实验室);East China Normal University(华东师范大学)
Unlocking Patch-Level Features for CLIP-Based Class-Incremental Learning
School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);State Key Laboratory for Novel Software Technology, Nanjing University(南京大学软件新技术国家重点实验室)
TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation
ARC Lab, Tencent PCG(腾讯PCG广告实验室);City University of Hong Kong(香港城市大学);Zhejiang University(浙江大学);NLPR & MAIS, Institute of Automation, CAS, Beijing(自动化研究所)
GMGaze: MoE-Based Context-Aware Gaze Estimation with CLIP and Multiscale Transformer
School of Artificial Intelligence, Guilin University of Electronic Technology, Guilin, China(桂林电子科技大学人工智能学院);The Laboratory for Imagery, Vision and Artificial Intelligence, École de Technologie Supérieure, Montreal, Canada(图像、视觉与人工智能实验室,蒙特利尔大学高级技术学院);Information Technology Department, College of Computer and Information Sciences, King Saud University, Riyadh, Saudi Arabia(计算机与信息科学学院信息科技系,沙特国王大学);Department of Cybersecurity, College of Computer Science and Engineering, Taibah University, Medina, Saudi Arabia(计算机科学与工程学院网络安全系,塔伊卜大学);Department of Networked Engineering, College of Computer Science and Engineering, Jeddah University, Jeddah, Saudi Arabia(计算机科学与工程学院网络工程系,吉达大学)