TCP:Textual-based Class-aware Prompt tuning for Visual-Language Model
Institute of Automation, CAS（中国科学院自动化研究所）；State Key Laboratory of Multimodal Artificial Intelligence Systems（多模态人工智能系统国家重点实验室）；State Key Lab of Processors（处理器国家重点实验室）；Institute of Computing Technology, CAS（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Prompt tuning represents a valuable technique for adapting pre-trained visual-language models (VLM) to various downstream tasks. Recent advancements in CoOp-based methods propose a set of learnable domain-shared or image-conditional textual tokens to facilitate the generation of task-specific textual classifiers. However, those textual tokens have a limited generalization ability regarding unseen domains, as they cannot dynamically adjust to the distribution of testing classes. To tackle this issue, we present a novel Textual-based Class-aware Prompt tuning(TCP) that explicitly incorporates prior knowledge about classes to enhance their discriminability. The critical concept of TCP involves leveraging Textual Knowledge Embedding (TKE) to map the high generalizability of class-level textual knowledge into class-aware textual tokens. By seamlessly integrating these class-aware prompts into the Text Encoder, a dynamic class-aware classifier is generated to enhance discriminability for unseen domains. During inference, TKE dynamically generates class-aware prompts related to the unseen classes. Comprehensive evaluations demonstrate that TKE serves as a plug-and-play module effortlessly combinable with existing methods. Furthermore, TCP consistently achieves superior performance while demanding less training time. Code:https://github.com/htyao89/Textual-based_Class-aware_prompt_tuning/
Tsinghua University（清华大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Columbia University（哥伦比亚大学）
Recent advances in pre-training vision-language models like CLIP have shown great potential in learning transferable visual representations. Nonetheless, for downstream inference, CLIP-like models suffer from either 1) degraded accuracy and robustness in the case of inaccurate text descriptions during retrieval-based inference (the challenge for zero-shot protocol); or 2) breaking the well-established vision-language alignment (the challenge for linear probing). To address them, we propose Decomposed Feature Prompting (DeFo). DeFo leverages a flexible number of learnable embeddings as textual …
Contrastive Language-Image Learning with Augmented Textual Prompts for 3D/4D FER Using Vision-Language Model
Information & Computer Science Department, King Fahd University of Petroleum & Minerals（国王法赫德石油与矿物大学信息与计算机科学系）；Center for Machine Vision and Signal Analysis, University of Oulu（奥卢大学机器视觉与信号分析中心）
Institute of Automation, CAS（中国科学院自动化研究所）；Institute of Computing Technology, CAS（中国科学院计算技术研究所）；University of Chinese Academy of Sciences(CAS)（中国科学院大学）
Prompt tuning is an effective way to adapt the pre-trained visual-language model (VLM) to the downstream task using task-related textual tokens. Representative CoOp-based work combines the learnable textual tokens with the class tokens to obtain specific textual knowledge. However, the specific textual knowledge is the worse generalization to the unseen classes because it forgets the essential general textual knowledge having a strong generalization ability. To tackle this issue, we introduce a novel Knowledge-guided Context Optimization (KgCoOp) to enhance the generalization ability of the le…
FedMVP: Federated Multimodal Visual Prompt Tuning for Vision-Language Models
University of Trento（特伦托大学）；University of Bergamo（贝拉姆奥大学）；Indian Institute of Technology Bombay（印度班加罗尔理工学院）；LNMIIT Jaipur（斋普尔LNMIIT）；The University of Queensland（昆士兰大学）；Fondazione Bruno Kessler（布鲁诺·凯塞勒基金会）
A-TPT: Angular Diversity Calibration Properties for Test-Time Prompt Tuning of Vision-Language Models
Dept. of Electronic and Telecommunication Engineering, University of Moratuwa（摩图瓦大学电子与电信工程系）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
A Similarity Paradigm Through Textual Regularization Without Forgetting
Shanghai Jiao Tong University（上海交通大学）；Hong Kong Baptist University（香港浸会大学）；Northeastern University（东北大学）；University of Science and Technology of China（中国科学技术大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
TextGaze: Prompting Gaze Target Estimation with Textual Scene Cues
University of Science and Technology of China（中国科学技术大学）；Hefei University of Technology（合肥工业大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；Anhui University（安徽大学）；United Arab Emirates University（阿联酋大学）
Defect-aware Hybrid Prompt Optimization via Progressive Tuning for Zero-Shot Multi-type Anomaly Detection and Segmentation
Corporate Research, Robert Bosch GmbH（罗伯特·博世集团企业研究部）；Otto-von-Guericke-University（奥托·冯·格里克大学）；Institute for Artificial Intelligence, University of Stuttgart（斯图加特大学人工智能研究所）；Faculty of Computer Science, UniVie Doctoral School Computer Science, University of Vienna（维也纳大学计算机科学系）；University of Southampton（南安普顿大学）
Render-of-Thought: Rendering Textual Chain-of-Thought as Images for Visual Latent Reasoning
Tencent BAC（腾讯BAC）；Shenzhen International Graduate School, Tsinghua University（深圳国际研究生院，清华大学）；School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）；School of Mathematics and Statistics, University of Glasgow（格拉斯哥大学数学与统计学学院）
We propose MM-REACT, a system paradigm that integrates ChatGPT with a pool of vision experts to achieve multimodal reasoning and action. In this paper, we define and explore a comprehensive list of advanced vision tasks that are intriguing to solve, but may exceed the capabilities of existing vision and vision-language models. To achieve such advanced visual intelligence, MM-REACT introduces a textual prompt design that can represent text descriptions, textualized spatial coordinates, and aligned file names for dense visual signals such as images and videos. MM-REACT's prompt design allows lan…
Continual Learning on CLIP via Incremental Prompt Tuning with Intrinsic Textual Anchors
School of Computer Science and Engineering, University of New South Wales（新南威尔士大学计算机科学与工程学院）；Data61, CSIRO（CSIRO数据61研究中心）；School of Computer Science, University of Auckland（奥克兰大学计算机科学学院）；Australian Institute for Machine Learning (AIML), The University of Adelaide（阿德莱德大学澳大利亚机器学习研究所）
Dynamic Textual Prompt For Rehearsal-free Lifelong Person Re-identification
National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology（空天地海一体化大数据应用技术国家工程实验室）；School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）
Prompt-driven Transferable Adversarial Attack on Person Re-Identification with Attribute-aware Textual Inversion
School of Artificial Intelligence and Robotics, Hunan University（湖南大学人工智能与机器人学院）；National Engineering Research Center of Robot Visual Perception and Control Technology（机器人视觉感知与控制技术国家工程研究中心）；College of Information Science and Engineering, Hunan Normal University（湖南师范大学信息科学与工程学院）
Jiangxi Normal University（江西师范大学）；University of Science and Technology of China（中国科学技术大学）；East China Normal University（华东师范大学）；Beijing Jiaotong University（北京交通大学）
MergeSlide: Continual Model Merging and Task-to-Class Prompt-Aligned Inference for Lifelong Learning on Whole Slide Images
Nara Institute of Science and Technology（奈良科学技术研究所）；Graduate School of Data Science, Chonnam National University（全南国立大学数据科学研究生院）；University of Information Technology, Viet Nam National University Ho Chi Minh City（越南胡志明市国家大学信息科技大学）；ETIS (UMR 8051), CY Cergy Paris University, ENSEA, CNRS（ETIS（UMR 8051），CY巴黎中央大学，ENSEA，CNRS）
Medal S: Spatio-Textual Prompt Model for Medical Segmentation
Medical Image Insights, Shanghai, China（医学影像洞察，上海，中国）；University of Washington, Seattle, WA, USA（华盛顿大学，西雅图，华盛顿州，美国）；University of Waterloo, Waterloo, ON, Canada（滑铁卢大学，滑铁卢，安大略省，加拿大）；Xi'an Jiaotong University, Xi'an, China（西安交通大学，西安，中国）
Visual and Textual Prompts in VLLMs for Enhancing Emotion Recognition
School of Computing, Australian National University (ANU)（澳大利亚国立大学计算机学院）；Quriosity Pty Ltd；Human-Centric Advancements Chair in AI, Curtin University and Australian National University（Curtin大学人工智能人本发展主席职位，澳大利亚国立大学）
TokenCLIP: Token-wise Prompt Learning for Zero-shot Anomaly Detection
Zhejiang University（浙江大学）；Tencent YouTu Lab（腾讯YouTu实验室）；Singapore Management University（新加坡管理学院）；Qilu University of Technology (Shandong Academy of Sciences)（青岛科技大学（山东科学院））
