Representation Calibration and Uncertainty Guidance for Class-Incremental Learning based on Vision Language Model
Guangdong Province Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education（广东省机器智能与先进计算重点实验室）；Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Peng Cheng Laboratory（鹏城实验室）；Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education（广东省机器智能与先进计算重点实验室）
Jiantao Tan, Peixian Ma, Tong Yu, Wentao Zhang, Ruixuan Wang
Class-incremental learning requires a learning system to continually learn knowledge of new classes and meanwhile try to preserve previously learned knowledge of old classes. As current state-of-the-art methods based on Vision-Language Models (VLMs) still suffer from the issue of differentiating classes across learning tasks. Here a novel VLM-based continual learning framework for image classification is proposed. In this framework, task-specific adapters are added to the pre-trained and frozen image encoder to learn new knowledge, and a novel cross-task representation calibration strategy based on a mixture of light-weight projectors is used to help better separate all learned classes in a unified feature space, alleviating class confusion across tasks. In addition, a novel inference strategy guided by prediction uncertainty is developed to more accurately select the most appropriate image feature for class prediction. Extensive experiments on multiple datasets under various settings demonstrate the superior performance of our method compared to existing ones.
DELTAMEM: Incremental Experience Memory for LLM Agents via Residual Trees
Beijing Key Laboratory of Research on Large Models and Intelligent Governance（北京大模型与智能治理重点实验室）；Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE（下一代智能搜索与推荐工程技术研究中心，教育部）；Gaoling School of Artificial, Renmin University of China（中国人民大学人工智能学院）；Duke University School of Medicine（杜克大学医学学院）
MPI for Informatics（马克斯·普朗克信息学研究所）；Goethe University Frankfurt（法兰克福大学）；MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）
Both generalized and incremental few-shot learning have to deal with three major challenges: learning novel classes from only few samples per class, preventing catastrophic forgetting of base classes, and classifier calibration across novel and base classes. In this work we propose a three-stage framework that allows to explicitly and effectively address these challenges. While the first phase learns base classes with many samples, the second phase learns a calibrated classifier for novel classes from few samples while also preventing catastrophic forgetting. In the final phase, calibration is…
Unlocking Patch-Level Features for CLIP-Based Class-Incremental Learning
School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；State Key Laboratory for Novel Software Technology, Nanjing University（南京大学软件新技术国家重点实验室）
Camera calibration plays a critical role in various computer vision tasks such as autonomous driving or augmented reality. Widely used camera calibration tools utilize plane pattern based methodology, such as using a chessboard or AprilTag board, user's calibration expertise level significantly affects calibration accuracy and consistency when without clear instruction. Furthermore, calibration is a recurring task that has to be performed each time the camera is changed or moved. It's also a great burden to calibrate huge amounts of cameras such as Driver Monitoring System (DMS) cameras in a p…
Cs2K: Class-specific and Class-shared Knowledge Guidance for Incremental Semantic Segmentation
Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences（中国科学院机器人与智能制造研究院）；University of Chinese Academy of Sciences（中国科学院大学）；College of Automation Science and Engineering, South China University of Technology（华南理工大学自动化科学与工程学院）；Peking University（北京大学）
Deep learning approaches are successful in a wide range of AI problems and in particular for visual recognition tasks. However, there are still open problems among which is the capacity to handle streams of visual information and the management of class imbalance in datasets. Existing research approaches these two problems separately while they co-occur in real world applications. Here, we study the problem of learning incrementally from imbalanced datasets. We focus on algorithms which have a constant deep model complexity and use a bounded memory to store exemplars of old classes across incr…
University of Chinese Academy of Sciences（中国科学院大学）；Huawei Inc.（华为公司）
Few-shot class-incremental learning (FSCIL) faces challenges of memorizing old class distributions and estimating new class distributions given few training samples. In this study, we propose a learnable distribution calibration (LDC) approach, with the aim to systematically solve these two challenges using a unified framework. LDC is built upon a parameterized calibration unit (PCU), which initializes biased distributions for all classes based on classifier vectors (memory-free) and a single covariance matrix. The covariance matrix is shared by all classes, so that the memory costs are fixed.…
Preserving Cross-Modal Consistency for CLIP-based Class-Incremental Learning
Institute of Trustworthy Embodied AI（可信具身人工智能研究院）；Fudan University（复旦大学）；Shanghai Collaborative Innovation Center of Intelligent Visual Computing（上海智能视觉计算协同创新中心）；Columbia University（哥伦比亚大学）
MVC-Bench: Benchmarking Calibration of Medical Vision-Language Models
Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Sabaragamuwa University of Sri Lanka（斯里兰卡萨巴拉加穆瓦大学）；Digital Platform Development, SLT PLC（SLT公共有限公司数字平台开发部）
ETH Zurich（苏黎世联邦理工学院）；INRIA Grenoble – Rhône-Alpes（法国国立信息学与自动化研究院格勒诺布尔-罗讷-阿尔卑斯分院）
It is well known that the accuracy of a calibration depends strongly on the choice of camera poses from which images of a calibration object are acquired. We present a system -- Calibration Wizard -- that interactively guides a user towards taking optimal calibration images. For each new image to be taken, the system computes, from all previously acquired images, the pose that leads to the globally maximum reduction of expected uncertainty on intrinsic parameters and then guides the user towards that pose. We also show how to incorporate uncertainty in corner point position in a novel principl…
Sculpting Margin Penalty: Intra-Task Adapter Merging and Classifier Calibration for Few-Shot Class-Incremental Learning
School of Computer Science and Technology, Beijing Institute of Technology（计算机科学与技术学院，北京理工大学）；School of Optics and Photonics, Beijing Institute of Technology（光学与 photonics 学院，北京理工大学）；School of Medical Technology, Beijing Institute of Technology（医学技术学院，北京理工大学）
Ambiguity-Guided Learnable Distribution Calibration for Semi-Supervised Few-Shot Class-Incremental Learning
New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别新实验室）；School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University（上海交通大学电子信息与电气工程学院）；School of Electronics and Information Engineering, Suzhou University of Science and Technology（苏州科技大学电子与信息工程学院）；Jiangsu Industrial Intelligent Low Carbon Technology Engineering Center（江苏省工业智能低碳技术工程中心）
Optimal Pose Guidance for Stereo Calibration in 3D Deformation Measurement
College of Aerospace Science and Engineering（航空航天科学与工程学院）；National University of Defense Technology（国防科技大学）；Hunan Provincial Key Laboratory of Image Measurement and Vision Navigation（湖南省图像测量与视觉导航重点实验室）；Hypersonic Technology Laboratory（高超声速技术实验室）；Hunan Key Laboratory of Intelligent Planning and Simulation for Aerospace Missions（湖南省航空航天任务智能规划与模拟重点实验室）
Consistency-Driven Calibration and Matching for Few-Shot Class-Incremental Learning
School of Automation, Central South University, China（中南大学自动化学院）；Big Data Institute, Central South University, China（中南大学大数据研究院）；School of Computer Science, Faculty of Engineering, The University of Sydney, Australia（悉尼大学工程学院计算机科学系）
VinUniversity（温纳大学）；University of Science, VNU-HCM（越南国家大学胡志明市分校科学大学）；Hanoi University of Science and Technology（河内理工大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Classifying pill categories from real-world images is crucial for various smart healthcare applications. Although existing approaches in image classification might achieve a good performance on fixed pill categories, they fail to handle novel instances of pill categories that are frequently presented to the learning algorithm. To this end, a trivial solution is to train the model with novel classes. However, this may result in a phenomenon known as catastrophic forgetting, in which the system forgets what it learned in previous classes. In this paper, we address this challenge by introducing t…
National University of Singapore（新加坡国立大学）；NUSRI Suzhou（新加坡国立大学苏州研究院）；Singapore General Hospital（新加坡中央医院）；The Chinese University of Hong Kong（香港中文大学）；Imperial College London（伦敦帝国理工学院）
Generating surgical reports aimed at surgical scene understanding in robot-assisted surgery can contribute to documenting entry tasks and post-operative analysis. Despite the impressive outcome, the deep learning model degrades the performance when applied to different domains encountering domain shifts. In addition, there are new instruments and variations in surgical tissues appeared in robotic surgery. In this work, we propose class-incremental domain adaptation (CIDA) with a multi-layer transformer-based model to tackle the new classes and domain shift in the target domain to generate surg…
Although recent deep learning-based calibration methods can predict extrinsic and intrinsic camera parameters from a single image, their generalization remains limited by the number and distribution of training data samples. The huge computational and space requirement prevents convolutional neural networks (CNNs) from being implemented in resource-constrained environments. This challenge motivated us to learn a CNN gradually, by training new data while maintaining performance on previously learned data. Our approach builds upon a CNN architecture to automatically estimate camera parameters (f…
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Nanjing University of Science and Technology（南京理工大学）；University of Leicester（莱斯特大学）
Object detection is one of the most fundamental yet challenging research topics in the domain of computer vision. Recently, the study on this topic in aerial images has made tremendous progress. However, complex background and worse imaging quality are obvious problems in aerial object detection. Most state-of-the-art approaches tend to develop elaborate attention mechanisms for the space-time feature calibrations with arduous computational complexity, while surprisingly ignoring the importance of feature calibrations in channel-wise. In this work, we propose a simple yet effective Calibrated-…
Nanjing University（南京大学）；Hikvision Research Institute（海康威视研究院）
New classes arise frequently in our ever-changing world, e.g., emerging topics in social media and new types of products in e-commerce. A model should recognize new classes and meanwhile maintain discriminability over old classes. Under severe circumstances, only limited novel instances are available to incrementally update the model. The task of recognizing few-shot new classes without forgetting old classes is called few-shot class-incremental learning (FSCIL). In this work, we propose a new paradigm for FSCIL based on meta-learning by LearnIng Multi-phase Incremental Tasks (LIMIT), which sy…
Gradient-Semantic Compensation for Incremental Semantic Segmentation
Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences（中国科学院机器人与智能制造创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Nanyang Technological University（南洋理工大学）
We address the problem of incremental semantic segmentation (ISS) recognizing novel object/stuff categories continually without forgetting previous ones that have been learned. The catastrophic forgetting problem is particularly severe in ISS, since pixel-level ground-truth labels are available only for the novel categories at training time. To address the problem, regularization-based methods exploit probability calibration techniques to learn semantic information from unlabeled pixels. While such techniques are effective, there is still a lack of theoretical understanding of them. Replay-bas…
I3DOD: Towards Incremental 3D Object Detection via Prompting
State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所机器人学国家重点实验室）；Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences（中国科学院机器人与智能制造研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Microsystem and Information Technology, Chinese Academy of Sciences（中国科学院微系统与信息技术研究所）
VinAI Research（VinAI研究院）；Oregon State University（俄勒冈州立大学）
This paper addresses incremental few-shot instance segmentation, where a few examples of new object classes arrive when access to training examples of old classes is not available anymore, and the goal is to perform well on both old and new classes. We make two contributions by extending the common Mask-RCNN framework in its second stage -- namely, we specify a new object class classifier based on the probit function and a new uncertainty-guided bounding-box predictor. The former leverages Bayesian learning to address a paucity of training examples of new classes. The latter learns not only to…
Incremental Learning (IL) is useful when artificial systems need to deal with streams of data and do not have access to all data at all times. The most challenging setting requires a constant complexity of the deep model and an incremental model update without access to a bounded memory of past data. Then, the representations of past classes are strongly affected by catastrophic forgetting. To mitigate its negative effect, an adapted fine tuning which includes knowledge distillation is usually deployed. We propose a different approach based on a vanilla fine tuning backbone. It leverages initi…
Most existing approaches to train a unified multi-organ segmentation model from several single-organ datasets require simultaneously access multiple datasets during training. In the real scenarios, due to privacy and ethics concerns, the training data of the organs of interest may not be publicly available. To this end, we investigate a data-free incremental organ segmentation scenario and propose a novel incremental training framework to solve it. We use the pretrained model instead of its own training data for privacy protection. Specifically, given a pretrained $K$ organ segmentation model …
Guidance Matters: Rethinking the Evaluation Pitfall for Text-to-Image Generation
The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Cogniser Information Technology（Cogniser信息科技）
Optimization-Based Calibration for Intravascular Ultrasound Volume Reconstruction
Inria；University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；INSERM（法国国家医学研究院）；ICube（ICube研究中心）；UMR7357（UMR7357研究单位）；University of Basel（巴塞尔大学）；Department of Biomedical Engineering（生物医学工程系）；Institute of Image-Guided Surgery（影像引导手术研究所）；IHU Strasbourg（斯特拉斯堡IHU）
Faculty of Mathematics and Informatics, Babeș-Bolyai University of Cluj-Napoca（巴比什-波雅依大学数学与信息学院）；Robert Bosch SRL（罗伯特·博世有限责任公司）
Most distortion correction methods focus on simple forms of distortion, such as radial or linear distortions. These works undistort images either based on measurements in the presence of a calibration grid, or use multiple views to find point correspondences and predict distortion parameters. When possible distortions are more complex, e.g. in the case of a camera being placed behind a refractive surface such as glass, the standard method is to use a calibration grid. Considering a high variety of distortions, it is nonviable to conduct these measurements. In this work, we present a single vie…
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人社会研究院）；University of California, Irvine（加利福尼亚大学尔湾分校）
The end-to-end Human Mesh Recovery (HMR) approach has been successfully used for 3D body reconstruction. However, most HMR-based frameworks reconstruct human body by directly learning mesh parameters from images or videos, while lacking explicit guidance of 3D human pose in visual data. As a result, the generated mesh often exhibits incorrect pose for complex activities. To tackle this problem, we propose to exploit 3D pose to calibrate human mesh. Specifically, we develop two novel Pose Calibration frameworks, i.e., Serial PC-HMR and Parallel PC-HMR. By coupling advanced 3D pose estimators an…
InfRS: Incremental Few-Shot Object Detection in Remote Sensing Images
School of Remote Sensing and Information Engineering, Wuhan University（武汉大学遥感信息工程学院）；Electronic Information School, Wuhan University（武汉大学电子信息学院）；Electrical and Computer Engineering, King Abdullah University of Science and Technology（阿卜杜拉国王科技大学电气与计算机工程学院）
Diffusion Model Meets Non-Exemplar Class-Incremental Learning and Beyond
Beijing National Research Center for Information Science and Technology (BNRist)（北京信息科学与技术国家研究中心（BNRist））；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）
EndoCIL: A Class-Incremental Learning Framework for Endoscopic Image Classification
School of Software（软件学院）；Hefei University of Technology（合肥工业大学）；School of Engineering Medicine（工程医学学院）；Beijing Advanced Innovation Center on Biomedical Engineering（北京生物医学工程创新中心）；Beihang University（北京航空航天大学）
