Open-vocabulary models are a promising new paradigm for image classification. Unlike traditional classification models, open-vocabulary models classify among any arbitrary set of categories specified with natural language during inference. This natural language, called "prompts", typically consists of a set of hand-written templates (e.g., "a photo of a {}") which are completed with each of the category names. This work introduces a simple method to generate higher accuracy prompts, without relying on any explicit knowledge of the task domain and with far fewer hand-constructed sentences. To achieve this, we combine open-vocabulary models with large language models (LLMs) to create Customized Prompts via Language models (CuPL, pronounced "couple"). In particular, we leverage the knowledge contained in LLMs in order to generate many descriptive sentences that contain important discriminating characteristics of the image categories. This allows the model to place a greater importance on these regions in the image when making predictions. We find that this straightforward and general approach improves accuracy on a range of zero-shot image classification benchmarks, including over one percentage point gain on ImageNet. Finally, this simple baseline requires no additional training and remains completely zero-shot. Code available at https://github.com/sarahpratt/CuPL.
TokenCLIP: Token-wise Prompt Learning for Zero-shot Anomaly Detection
Zhejiang University（浙江大学）；Tencent YouTu Lab（腾讯YouTu实验室）；Singapore Management University（新加坡管理学院）；Qilu University of Technology (Shandong Academy of Sciences)（青岛科技大学（山东科学院））
VideoMaker: Zero-shot Customized Video Generation with the Inherent Force of Video Diffusion Models
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；ARC Lab, Tencent PCG（腾讯平台与内容事业群ARC实验室）；Tencent AI Lab（腾讯AI实验室）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
Despite excellent progress has been made, the performance of deep learning based algorithms still heavily rely on specific datasets, which are difficult to extend due to labor-intensive labeling. Moreover, because of the advancement of new applications, initial definition of data annotations might not always meet the requirements of new functionalities. Thus, there is always a great demand in customized data annotations. To address the above issues, we propose the Few-Shot Model Adaptation (FSMA) framework and demonstrate its potential on several important tasks on Faces. The FSMA first acquir…
MAVIN: Multi-Shot Audio-Visual Generation with Customized Narrative Control
Peking University（北京大学）；Kling Team, Kuaishou Technology（快手团队）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Sun Yat-sen University（中山大学）
In-Context Brush: Zero-shot Customized Subject Insertion with Context-Aware Latent Space Manipulation
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；University of Konstanz（康斯坦茨大学）；National Cheng-Kung University（国立成功大学）
University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；Microsoft（微软）
Image-text contrastive learning models such as CLIP have demonstrated strong task transfer ability. The high generality and usability of these visual models is achieved via a web-scale data collection process to ensure broad concept coverage, followed by expensive pre-training to feed all the knowledge into model weights. Alternatively, we propose REACT, REtrieval-Augmented CusTomization, a framework to acquire the relevant web knowledge to build customized visual models for target domains. We retrieve the most relevant image-text pairs (~3% of CLIP pre-training data) from the web-scale databa…
StoryAgent: Customized Storytelling Video Generation via Multi-Agent Collaboration
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；United Arab Emirates University（阿拉伯联合酋长国大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
Linking heterogeneous microstructure informatics with expert characterization knowledge through customized and hybrid vision-language representations for industrial qualification
McGill University（麦吉尔大学）；National Research Council Canada（加拿大国家研究委员会）；Apollo-Clad Laser Cladding, a division of Apollo Machine and Welding Ltd.（Apollo-Clad激光熔覆，Apollo机械与焊接有限公司旗下部门）；Fraunhofer Institute for Laser Technology ILT（弗劳恩霍夫激光技术研究所）
The existing few-shot video classification methods often employ a meta-learning paradigm by designing customized temporal alignment module for similarity calculation. While significant progress has been made, these methods fail to focus on learning effective representations, and heavily rely on the ImageNet pre-training, which might be unreasonable for the few-shot recognition setting due to semantics overlap. In this paper, we aim to present an in-depth study on few-shot video classification by making three contributions. First, we perform a consistent comparative study on the existing metric…
Towards General Text-guided Image Synthesis for Customized Multimodal Brain MRI Generation
ShanghaiTech University（上海科技大学）；Shanghai Clinical Research and Trial Center（上海临床研究与试验中心）；Affiliated Hangzhou First People’s Hospital, Xihu University School of Medicine（西湖大学医学院附属杭州市第一人民医院）
PhotoDoodle: Learning Artistic Image Editing from Few-Shot Pairwise Data
National University of Singapore（新加坡国立大学）；Shanghai Jiao Tong University（上海交通大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Tiamat（提亚马特）
One-to-Normal: Anomaly Personalization for Few-shot Anomaly Detection
West China Hospital, Sichuan University（四川大学华西医院）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Sichuan University Pittsburgh Institute, Sichuan University（四川大学匹兹堡学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Few-shot Learner Parameterization by Diffusion Time-steps
Nanyang Technological University（南洋理工大学）；Sea AI Lab（Sea AI实验室）；Singapore Management University（新加坡管理大学）；Hefei University of Technology（合肥工业大学）；Skywork AI
One-Shot Novel View and Pose Human Image Synthesis via 3D Prior Guided Diffusion Model
PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, and Jiangsu Key Lab of Image and Video Understanding for Social Security, School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院教育部高维信息智能感知与系统重点实验室、江苏省社会安全图像与视频理解重点实验室及PCA实验室）；Advanced Laser Technology Laboratory of Anhui Province, Electronic Engineering Institute, National University of Defense Technology, and Jianghuai Advance Technology Center（国防科技大学电子工程学院安徽省先进激光技术实验室及江淮前沿技术中心）
MuseumMaker: Continual Style Customization without Catastrophic Forgetting
State Key Laboratory of Robotics, the Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences（中国科学院机器人与智能制造研究院机器人国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；School of Automation Science and Engineering, South China University of Technology（华南理工大学自动化科学与工程学院）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Salesforce AI Research（Salesforce AI研究部门）
FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；ByteDance Inc（字节跳动公司）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
