Prompt learning has become a prevalent strategy for adapting vision-language foundation models to downstream tasks. As large language models (LLMs) have emerged, recent studies have explored the use of category-related descriptions as input to enhance prompt effectiveness. Nevertheless, conventional descriptions fall short of structured information that effectively represents the interconnections among entities or attributes linked to a particular category. To address this limitation and prioritize harnessing structured knowledge, this paper advocates for leveraging LLMs to build a graph for each description to model the entities and attributes describing the category, as well as their correlations. Preexisting prompt tuning methods exhibit inadequacies in managing this structured knowledge. Consequently, we propose a novel approach called Hierarchical Prompt Tuning (HPT), which enables simultaneous modeling of both structured and conventional linguistic knowledge. Specifically, we introduce a relationship-guided attention module to capture pair-wise associations among entities and attributes for low-level prompt learning. In addition, by incorporating high-level and global-level prompts modeling overall semantics, the proposed hierarchical structure forges cross-level interlinks and empowers the model to handle more complex and long-term relationships. Extensive experiments demonstrate that our HPT shows strong effectiveness and generalizes much better than existing SOTA methods. Our code is available at https://github.com/Vill-Lab/2024-AAAI-HPT.
PVLR: Prompt-driven Visual-Linguistic Representation Learning for Multi-Label Image Recognition
MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所MAIS）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Baidu Inc.（百度公司）；CAIR, HKISI, Chinese Academy of Sciences（中国科学院香港中文大学深圳先进集成技术研究所CAIR）
University of Chinese Academy of Science（中国科学院大学）；Australian Institute for Machine Learning, The University of Adelaide（阿德莱德大学澳大利亚机器学习研究所）；Peng Cheng Laboratory（鹏城实验室）；University of California, Merced（加州大学默塞德分校）
Video captioning aims to generate natural language descriptions according to the content, where representation learning plays a crucial role. Existing methods are mainly developed within the supervised learning framework via word-by-word comparison of the generated caption against the ground-truth text without fully exploiting linguistic semantics. In this work, we propose a hierarchical modular network to bridge video representations and linguistic semantics from three levels before generating captions. In particular, the hierarchy is composed of: (I) Entity level, which highlights objects th…
Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning
Hacettepe University（哈杰泰佩大学）；Aselsan Research（阿塞尔桑研究院）；Koç University（科克大学）；Amsterdam UMC, University of Amsterdam（阿姆斯特丹大学医学中心，阿姆斯特丹大学）；Amsterdam Public Health（阿姆斯特丹公共卫生研究院）
Tone Matters: The Impact of Linguistic Tone on Hallucination in VLMs
Department of Computer Science and Technology, Kean University（计算机科学与技术系，凯恩大学）；Department of Computer Science and Engineering, University of Notre Dame（计算机科学与工程系，圣母大学）
Modern convolutional neural networks (CNNs) are able to achieve human-level object classification accuracy on specific tasks, and currently outperform competing models in explaining complex human visual representations. However, the categorization problem is posed differently for these networks than for humans: the accuracy of these networks is evaluated by their ability to identify single labels assigned to each image. These labels often cut arbitrarily across natural psychological taxonomies (e.g., dogs are separated into breeds, but never jointly categorized as "dogs"), and bias the resulti…
Hierarchical Self-Prompting SAM: A Prompt-Free Medical Image Segmentation Framework
State Key Laboratory of Multimodal Artificial Intelligence System, Institute of Automation, Chinese Academy of Science, China（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院，中国）；School of Artificial Intelligence, University of Chinese Academy of Science, China（人工智能学院，中国科学院大学，中国）；Department of Radiology, Peking Union Medical College Hospital, Peking Union Medical College, Chinese Academy of Medical Sciences, Beijing（放射科，北医三院，北京医科大学，中国医学科学院，北京）
Linguistic Profiling of Deepfakes: An Open Database for Next-Generation Deepfake Detection
University of Southampton（南安普顿大学）；Harbin Institute of Technology（哈尔滨工业大学）；Pengcheng Laboratory（鹏城实验室）；Shenzhen Institute of Advanced Technology, Chinese Academy of Science（中国科学院深圳先进技术研究院）
HiPrompt: Tuning-free Higher-Resolution Generation with Hierarchical MLLM Prompts
Hong Kong University of Science and Technology（香港科技大学）；Nanyang Technological University（南洋理工大学）；Tsinghua University（清华大学）；University of Chinese Academy of Sciences（中国科学院大学）
Hierarchical Prompts for Rehearsal-free Continual Learning
University of Science and Technology of China（中国科学技术大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Tianjin University of Technology（天津理工大学）；University of Chinese Academy of Sciences（中国科学院大学）
Hierarchical Visual Prompt Learning for Continual Video Instance Segmentation
Mohamed bin Zayed University of Artificial Intelligence（莫扎德·本·扎耶德人工智能大学）；Hunan University（湖南大学）；University of Trento（特伦托大学）；Zhejiang University（浙江大学）；Fudan University（复旦大学）；Australian National University（澳大利亚国立大学）；Linköping University（利乌波尔特大学）
FlowCred: Hierarchical Credit Assignment and Functional Coordination for Joint Text-to-Video Post-Training
Shanghai Jiao Tong University（上海交通大学）；KlingAI；The Hong Kong University of Science and Technology（香港科技大学）；South China University of Technology（华南理工大学）；The University of Hong Kong（香港大学）
FaVChat: Hierarchical Prompt-Query Guided Facial Video Understanding with Data-Efficient GRPO
State Key Laboratory of Networking and Switching Technology, BUPT（北京邮电大学网络与交换技术国家重点实验室）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东人工智能与数字经济实验室（深圳））；Institute of automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Zhongguancun Academy（中关村学院）；Central South University（中南大学）；Zhejiang University（浙江大学）；College of communication Engineering, Hangzhou Dianzi University（杭州电子科技大学通信工程学院）
Dual-View Alignment Learning with Hierarchical-Prompt for Class-Imbalance Multi-Label Classification
Ministry of Education Key Laboratory of Dependable Service Computing in Cyber Physical Society（教育部可信服务计算网络社会重点实验室）；School of Big Data and Software Engineering（大数据与软件工程学院）；School of Computer Science and Information Engineering（计算机科学与信息工程学院）
HCVP: Leveraging Hierarchical Contrastive Visual Prompt for Domain Generalization
Mohamed bin Zayed University of Artificial Intelligence（莫扎德·本·扎耶德人工智能大学）；University of New South Wales（新南威尔士大学）；Carnegie Mellon University（卡内基梅隆大学）；University of California, San Diego（加州大学圣地亚哥分校）；CSIRO’s Data61（CSIRO数据61）；University of Sydney（悉尼大学）；Macquarie University（麦考瑞大学）
