Soft prompt learning has recently emerged as one of the methods of choice for adapting V&L models to a downstream task using a few training examples. However, current methods significantly overfit the training data, suffering from large accuracy degradation when tested on unseen classes from the same domain. To this end, in this paper, we make the following 4 contributions: (1) To alleviate base class overfitting, we propose a novel Language-Aware Soft Prompting (LASP) learning method by means of a text-to-text cross-entropy loss that maximizes the probability of the learned prompts to be correctly classified with respect to pre-defined hand-crafted textual prompts. (2) To increase the representation capacity of the prompts, we propose grouped LASP where each group of prompts is optimized with respect to a separate subset of textual prompts. (3) We identify a visual-language misalignment introduced by prompt learning and LASP, and more importantly, propose a re-calibration mechanism to address it. (4) We show that LASP is inherently amenable to including, during training, virtual classes, i.e. class names for which no visual samples are available, further increasing the robustness of the learned prompts. Through evaluations on 11 datasets, we show that our approach (a) significantly outperforms all prior works on soft prompting, and (b) matches and surpasses, for the first time, the accuracy on novel classes obtained by hand-crafted prompts and CLIP for 8 out of 11 test datasets. Code will be made available at https://www.adrianbulat.com/lasp
PromptEmbedder: Efficient and Transferable Text Embedding via Dual-LLM Soft Prompting
Department of Computer Science and Information Engineering, National Taiwan University(国立台湾大学计算机科学与资讯工程系);National Taiwan University AI Center of Research Excellence(国立台湾大学人工智能研究中心)
Unlocking Memorization in Large Language Models with Dynamic Soft Prompting
George Mason University(乔治梅森大学);GE Healthcare(GE医疗);University of Pittsburgh(匹兹堡大学);William and Mary(威廉玛丽学院);Southern University of Science and Technology(南方科技大学);Florida State University(佛罗里达州立大学)
From PowerPoint UI Sketches to Web-Based Applications: Pattern-Driven Code Generation for GIS Dashboard Development Using Knowledge-Augmented LLMs, Context-Aware Visual Prompting, and the React Framework
An Empirical Evaluation of LLM-Generated Code Security Across Prompting Methods
Department of Computer Science, Birzeit University(计算机科学系,巴勒斯坦比泽大学);King Fahd University of Petroleum and Minerals(国王法赫德石油和矿物大学);University of Central Florida(中央佛罗里达大学)
Using LLMs in Generating Design Rationale for Software Architecture Decisions
School of Computer Science, Wuhan University(武汉大学计算机学院);RMIT University(皇家墨尔本理工大学);School of Computer Science, Central China Normal University(中央师范大学计算机学院);School of Artificial Intelligence, Shenzhen Polytechnic University(深圳职业技术学院人工智能学院)
SSPA: Split-and-Synthesize Prompting with Gated Alignments for Multi-Label Image Recognition
School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)(中国科学院大学人工智能学院);School of Computer Science and Engineering, Macau University of Science and Technology(澳门科技大学计算机科学与工程学院);Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences(中国科学院香港创新研究院人工智能与机器人中心);Sangfor Technologies Inc.(深信服科技股份有限公司);Shenzhen Institute of Advanced Technology (SIAT), Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Westlake University(西湖大学)
GRASP: Guided Region-Aware Sparse Prompting for Adapting MLLMs to Remote Sensing
School of Computing, Kyung Hee University(京畿大学计算机学院);School of Information and Software Engineering, University of Electronic Science and Technology of China(电子科技大学信息与软件工程学院);School of Computer Science and Engineering, University of Electronic Science and Technology of China(电子科技大学计算机科学与工程学院);College of Computer Science and Information Engineering, Harbin Normal University(哈尔滨师范大学计算机科学与信息工程学院);College of Professional and Continuing Education, The Hong Kong Polytechnic University(香港理工大学专业及继续教育学院);School of Computer Science and Technology, Tongji University(同济大学计算机科学与技术学院)
Prompt Engineering Strategies for LLM-based Qualitative Coding of Psychological Safety in Software Engineering Communities: A Controlled Empirical Study
Federal University of Bahia(巴伊亚联邦大学);State University of Feira de Santana(费拉德桑塔纳州立大学);Federal University of Sergipe(塞格皮联邦大学);Federal Institute of Ceara(塞阿拉联邦理工学院)
VRPTEST: Evaluating Visual Referring Prompting in Large Multimodal Models
Hong Kong University of Science and Technology(香港科技大学);Harbin Institute of Technology(哈尔滨工业大学);National University of Singapore(新加坡国立大学);Nanyang Technological University(南洋理工大学)
MAP-VLA: Memory-Augmented Prompting for Vision-Language-Action Model in Robotic Manipulation
Nanyang Technological University(南洋理工大学);VinUniversity(文大学);Beijing University of Posts and Telecommunications(北京邮电大学);Tsinghua University(清华大学);South China University of Technology(华南理工大学)