CLIP-Guided Adaptable Self-Supervised Learning for Human-Centric Visual Tasks
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS), and University of Chinese Academy of Sciences（智能信息处理重点实验室，计算技术研究所（ICT），中国科学院（CAS）及中国科学院大学）
Mingshuang Luo, Ruibing Hou, Bo Chao, Hong Chang, Zimo Liu, Yaowei Wang, Shiguang Shan
备注：
Accepted by TMM (IEEE Transactions on Multimedia), 16 pages, 7 figures
Human-centric visual analysis plays a pivotal role in diverse applications, including surveillance, healthcare, and human-computer interaction. With the emergence of large-scale unlabeled human image datasets, there is an increasing need for a general unsupervised pre-training model capable of supporting diverse human-centric downstream tasks. To achieve this goal, we propose CLASP (CLIP-guided Adaptable Self-suPervised learning), a novel framework designed for unsupervised pre-training in human-centric visual tasks. CLASP leverages the powerful vision-language model CLIP to generate both low-level (e.g., body parts) and high-level (e.g., attributes) semantic pseudo-labels. These multi-level semantic cues are then integrated into the learned visual representations, enriching their expressiveness and generalizability. Recognizing that different downstream tasks demand varying levels of semantic granularity, CLASP incorporates a Prompt-Controlled Mixture-of-Experts (MoE) module. MoE dynamically adapts feature extraction based on task-specific prompts, mitigating potential feature conflicts and enhancing transferability. Furthermore, CLASP employs a multi-task pre-training strategy, where part- and attribute-level pseudo-labels derived from CLIP guide the representation learning process. Extensive experiments across multiple benchmarks demonstrate that CLASP consistently outperforms existing unsupervised pre-training methods, advancing the field of human-centric visual analysis.
fine-CLIP: Enhancing Zero-Shot Fine-Grained Surgical Action Recognition with Vision-Language Models
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；INSERM（法国国家健康与医学研究院）；ICube（ICube实验室）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University Hospital of Strasbourg（斯特拉斯堡大学医院）
Proximity-CLIP: Text-Guided Semantic Proximity Learning for Zero-Shot Anomaly Detection
School of Software Engineering, Xi’an Jiaotong University（西安交通大学软件学院）；Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（西安交通大学人工智能与机器人研究所）
CLARA: Clip-Level Multimodal Alignment with VLM-Derived Rationales for Hateful Video Detection
Institute for Analytics and Data Science, University of Essex（埃塞克斯大学分析与数据科学研究所）；University of Exeter（埃克塞特大学）；Queen Mary University of London（伦敦大学玛丽皇后学院）
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Samsung AI Centre Toronto（三星多伦多人工智能中心）；Google Research（谷歌研究院）
Understanding dynamics from visual observations is a challenging problem that requires disentangling individual objects from the scene and learning their interactions. While recent object-centric models can successfully decompose a scene into objects, modeling their dynamics effectively still remains a challenge. We address this problem by introducing SlotFormer -- a Transformer-based autoregressive model operating on learned object-centric representations. Given a video clip, our approach reasons over object features to model spatio-temporal relationships and predicts accurate future object s…
Human Pose Descriptions and Subject-Focused Attention for Improved Zero-Shot Transfer in Human-Centric Classification Tasks
University of Kaiserslautern-Landau（凯泽斯劳滕-兰道大学）；Deutsches Forschungszentrum für Künstliche Intelligenz（德国人工智能研究中心）；ETH Zürich（苏黎世联邦理工学院）；Technical University of Munich（慕尼黑工业大学）
Person-Centric Annotations of LAION-400M: Auditing Bias and Its Transfer to Models
Technical University of Munich, Munich Center for Machine Learning (MCML), MDSI（慕尼黑工业大学，慕尼黑机器学习中心（MCML），MDSI）；LTCI, Télécom Paris, Institut Polytechnique de Paris, France（法国巴黎理工学院，巴黎电信学院，LTCI）；Helmholtz Munich（亥姆霍兹慕尼黑中心）；University of California, Berkeley（加州大学伯克利分校）
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Montreal Institute for Learning Algorithms（蒙特利尔学习算法研究所）
There is growing interest in artificial intelligence to build socially intelligent robots. This requires machines to have the ability to "read" people's emotions, motivations, and other factors that affect behavior. Towards this goal, we introduce a novel dataset called MovieGraphs which provides detailed, graph-based annotations of social situations depicted in movie clips. Each graph consists of several types of nodes, to capture who is present in the clip, their emotional and physical attributes, their relationships (i.e., parent/child), and the interactions between them. Most interactions …
We propose an effective two-stage approach to tackle the problem of language-based Human-centric Spatio-Temporal Video Grounding (HC-STVG) task. In the first stage, we propose an Augmented 2D Temporal Adjacent Network (Augmented 2D-TAN) to temporally ground the target moment corresponding to the given description. Primarily, we improve the original 2D-TAN from two aspects: First, a temporal context-aware Bi-LSTM Aggregation Module is developed to aggregate clip-level representations, replacing the original max-pooling. Second, we propose to employ Random Concatenation Augmentation (RCA) mechan…
3D Feature Distillation with Object-Centric Priors
Department of Artificial Intelligence University of Groningen, the Neteherlands（格罗宁根大学人工智能系）；School of Informatics University of Edinburgh, United Kingdom（爱丁堡大学信息学院）；Department of Computer Science University College London, United Kingdom（伦敦大学学院计算机科学系）
Australian National University（澳大利亚国立大学）；Linköping University（林雪平大学）；Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）
Existing open-vocabulary object detectors typically enlarge their vocabulary sizes by leveraging different forms of weak supervision. This helps generalize to novel objects at inference. Two popular forms of weak-supervision used in open-vocabulary detection (OVD) include pretrained CLIP model and image-level supervision. We note that both these modes of supervision are not optimally aligned for the detection task: CLIP is trained with image-text pairs and lacks precise localization of objects while the image-level supervision has been used with heuristics that do not accurately specify local …
DC-Scene: Data-Centric Learning for 3D Scene Understanding
Shanghai University of Engineering Science（上海工程科学大学）；The Australian National University（澳大利亚国立大学）；Sun Yat-sen University（中山大学）；La Trobe University（拉筹伯大学）
GreenRFM: Learning a resource-efficient radiology vision-language foundation model via supervision-centric pre-training
School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China (USTC)（生物医学工程学院，生命科学与医学系，中国科学技术大学）；Center for Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE)（医学影像、机器人、分析计算与学习中心）；Suzhou Institute for Advanced Research, USTC（苏州先进研究院，中国科学技术大学）；Department of Radiology, The First Affiliated Hospital of USTC, Division of Life Sciences and Medicine, USTC（放射科，中国科学技术大学第一附属医院，生命科学与医学系，中国科学技术大学）；T Magnetic Resonance Translational Medicine Research Center, Department of Radiology, The First Affiliated Hospital (Southwest Hospital) of Army Medical University（7T磁共振转化医学研究中心，放射科，中国医学大学第一附属医院（西南医院））
KU Leuven（鲁汶大学）；ShanghaiTech University（上海科技大学）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程技术研究中心）
Human object interaction (HOI) detection plays a crucial role in human-centric scene understanding and serves as a fundamental building-block for many vision tasks. One generalizable and scalable strategy for HOI detection is to use weak supervision, learning from image-level annotations only. This is inherently challenging due to ambiguous human-object associations, large search space of detecting HOIs and highly noisy training signal. A promising strategy to address those challenges is to exploit knowledge from large-scale pretrained models (e.g., CLIP), but a direct knowledge distillation s…
