AllInOne训练范式以多任务学习的方式将广泛的任务压缩到一个统一模型中。然而，多任务学习中的优化比单任务学习更具挑战性，因为来自不同任务的梯度范数可能差异很大，使得骨干网络过度偏向某一特定任务。为了解决这一问题，我们提出了任务级骨干导向梯度裁剪范式，与普通的梯度裁剪方法相比，它有两个重点：1）对每个任务独立执行梯度裁剪。2）将每个任务生成的骨干梯度重新缩放到相同的范数尺度。基于实验结果，我们认为任务级骨干导向梯度裁剪范式可以在一定程度上缓解梯度偏差问题。我们还提出了一种新颖的多分支数据增强策略，其中相互冲突的增强被放置在不同的分支中。我们的方法已被证明是有效的，并最终在CVPR2023 Foundation Model Challenge的Leaderboard A中获得第1名，在Leaderboard B中获得第2名。值得注意的是，Leaderboard A评估了所有三个任务（检测、分割和细粒度分类），而Leaderboard B不评估分割任务，我们团队在该任务上具有巨大优势。
英文摘要：
The AllInOne training paradigm squeezes a wide range of tasks into a unified model in a multi-task learning manner. However, optimization in multi-task learning is more challenge than single-task learning, as the gradient norm from different tasks may vary greatly, making the backbone overly biased towards one specific task. To address this issue, we propose the task-level backbone-oriented gradient clip paradigm, compared with the vanilla gradient clip method, it has two points of emphasis:1) gradient clip is performed independently for each task. 2) backbone gradients generated from each task are rescaled to the same norm scale. Based on the experimental results, we argue that the task-level backbone-oriented gradient clip paradigm can relieve the gradient bias problem to some extent. We also propose a novel multi-branch data augmentation strategy where conflict augmentations are placed in different branches. Our approach has been shown to be effective and finally achieve 1st place in the Leaderboard A and 2nd place in the Leaderboard B of the CVPR2023 Foundation Model Challenge. It's worth noting that instead of evaluating all three tasks(detection, segmentation and fine-grained classification) in Leaderboard A, the segmentation task is not evaluated in Leaderboard B, in which our team has a huge advantage.
From Global to Local: Rethinking CLIP Feature Aggregation for Person Re-Identification
Department of Electrical and Computer Engineering, University of Washington（华盛顿大学电气与计算机工程系）；Applied Physics Laboratory, University of Washington（华盛顿大学应用物理实验室）
Unveiling Backbone Effects in CLIP: Exploring Representational Synergies and Variances
Australian Institute for Machine Learning（澳大利亚机器学习研究所）；University of Adelaide（阿德莱德大学）；The University of Tokyo（东京大学）；AIST（日本产业技术综合研究所）；University of Chile（智利大学）；CENIA（智利国家人工智能研究中心）；IMFD（千年基础数据研究所）；Amazon（亚马逊）
ML-CLIPSim: Multi-Layer CLIP Similarity for Machine-Oriented Image Quality
Simon Fraser University（西蒙弗雷泽大学）；University of British Columbia（不列颠哥伦比亚大学）；Eastern Institute of Technology（东部技术学院）；Xi’an Jiaotong University（西安交通大学）；Google Inc（谷歌公司）
COSMIC: Clique-Oriented Semantic Multi-space Integration for Robust CLIP Test-Time Adaptation
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Shenzhen Technology University（深圳技术大学）；University of Science and Technology of China（中国科学技术大学）
We propose Spatio-temporal Crop Aggregation for video representation LEarning (SCALE), a novel method that enjoys high scalability at both training and inference time. Our model builds long-range video features by learning from sets of video clip-level features extracted with a pre-trained backbone. To train the model, we propose a self-supervised objective consisting of masked clip feature prediction. We apply sparsity to both the input, by extracting a random set of video clips, and to the loss function, by only reconstructing the sparse inputs. Moreover, we use dimensionality reduction by w…
NAVER Shopping（NAVER购物）；NAVER AI Research（NAVER人工智能研究院）
Understanding vision and language representations of product content is vital for search and recommendation applications in e-commerce. As a backbone for online shopping platforms and inspired by the recent success in representation learning research, we propose a contrastive learning framework that aligns language and visual models using unlabeled raw product text and images. We present techniques we used to train large-scale representation learning models and share solutions that address domain-specific challenges. We study the performance using our pre-trained model as backbones for diverse…
Many video instance segmentation (VIS) methods partition a video sequence into individual frames to detect and segment objects frame by frame. However, such a frame-in frame-out (FiFo) pipeline is ineffective to exploit the temporal information. Based on the fact that adjacent frames in a short clip are highly coherent in content, we propose to extend the one-stage FiFo framework to a clip-in clip-out (CiCo) one, which performs VIS clip by clip. Specifically, we stack FPN features of all frames in a short video clip to build a spatio-temporal feature cube, and replace the 2D conv layers in the…
Contrastive language image pretraining (CLIP) encoders have been shown to be beneficial for a range of visual tasks from classification and detection to captioning and image manipulation. We investigate the effectiveness of CLIP visual backbones for Embodied AI tasks. We build incredibly simple baselines, named EmbCLIP, with no task specific architectures, inductive biases (such as the use of semantic maps), auxiliary tasks during training, or depth maps -- yet we find that our improved baselines perform very well across a range of tasks and simulators. EmbCLIP tops the RoboTHOR ObjectNav lead…
MROVSeg: Breaking the Resolution Curse of Vision-Language Models in Open-Vocabulary Image Segmentation
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所基础模型研究中心）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）；Zhejiang Lab（之江实验室）；University of Chinese Academy of Sciences（中国科学院大学）
University of Modena and Reggio Emilia（摩德纳与雷焦艾米利亚大学）；METALIQUID S.R.L.（METALIQUID有限责任公司）
Action Detection is a complex task that aims to detect and classify human actions in video clips. Typically, it has been addressed by processing fine-grained features extracted from a video classification backbone. Recently, thanks to the robustness of object and people detectors, a deeper focus has been added on relationship modelling. Following this line, we propose a graph-based framework to learn high-level interactions between people and objects, in both space and time. In our formulation, spatio-temporal relationships are learned through self-attention on a multi-layer graph structure wh…
The University of Tokyo（东京大学）；Google Research（谷歌研究院）
Domain generalization (DG) is a difficult transfer learning problem aiming to learn a generalizable model for unseen domains. Recent foundation models (FMs) are robust to many distribution shifts and, therefore, should substantially improve the performance of DG. In this work, we study generic ways to adopt CLIP, a Visual-Language Foundation Model, for DG problems in image classification. While ERM greatly improves the accuracy with bigger backbones and training datasets using standard DG benchmarks, fine-tuning FMs is not practical in many real-world situations. We propose Domain Prompt Learn…
DeCLIP: Decoupled Prompting for Multi-Label Class-Incremental Learning with CLIP
School of Automation, Southeast University, China（东南大学自动化学院）；University of the Chinese Academy of Sciences, China（中国科学院大学）；Suzhou University of Science and Technology, China（苏州科技大学）；Computer Vision Center, Spain（西班牙计算机视觉中心）
Bridging the Task Gap: Multi-Task Adversarial Transferability in CLIP and Its Derivatives
Shenzhen Campus of Sun Yat-sen University, China（中山大学深圳校区）；The National University of Singapore, Singapore（新加坡国立大学）；National Key Laboratory of Science and Technology on Information System Security, China（信息系统安全科学技术国家重点实验室）
Multi-Perspective Subimage CLIP with Keyword Guidance for Remote Sensing Image-Text Retrieval
School of Computer Technology and Applications（计算机技术与应用学院）；Qinghai University（青海大学）；Qinghai Provincial Laboratory for Intelligent Computing and Application（青海省智能计算与应用实验室）
Describe, Adapt and Combine: Empowering CLIP Encoders for Open-set 3D Object Retrieval
Huazhong Agricultural University（华中农业大学）；Shenzhen University（深圳大学）；The University of Hong Kong（香港大学）；University of Louisville（路易斯安那大学）；ByteDance（字节跳动）；Huazhong University of Science and Technology（华中科技大学）
National Central University（国立中央大学）；University of Arkansas（阿肯色大学）
Self-supervised tasks have been utilized to build useful representations that can be used in downstream tasks when the annotation is unavailable. In this paper, we introduce a self-supervised video representation learning method based on the multi-transformation classification to efficiently classify human actions. Self-supervised learning on various transformations not only provides richer contextual information but also enables the visual representation more robust to the transforms. The spatio-temporal representation of the video is learned in a self-supervised manner by classifying seven d…
