Visual Prompting in LLMs for Enhancing Emotion Recognition
Australian National University(澳大利亚国立大学);Quriosity Pty Ltd(Quriosity私人有限公司);Webumate Pty Ltd(Webumate私人有限公司);Curtin University(科廷大学);Yale University(耶鲁大学)
Qixuan Zhang, Zhifeng Wang, Dylan Zhang, Wenjia Niu, Sabrina Caldwell, Tom Gedeon, Yang Liu, Zhenyue Qin
Vision Large Language Models (VLLMs) are transforming the intersection of computer vision and natural language processing. Nonetheless, the potential of using visual prompts for emotion recognition in these models remains largely unexplored and untapped. Traditional methods in VLLMs struggle with spatial localization and often discard valuable global context. To address this problem, we propose a Set-of-Vision prompting (SoV) approach that enhances zero-shot emotion recognition by using spatial information, such as bounding boxes and facial landmarks, to mark targets precisely. SoV improves accuracy in face count and emotion categorization while preserving the enriched image context. Through a battery of experimentation and analysis of recent commercial or open-source VLLMs, we evaluate the SoV model's ability to comprehend facial expressions in natural environments. Our findings demonstrate the effectiveness of integrating spatial visual prompts into VLLMs for improving emotion recognition performance.
Visual and Textual Prompts in VLLMs for Enhancing Emotion Recognition
School of Computing, Australian National University (ANU)(澳大利亚国立大学计算机学院);Quriosity Pty Ltd;Human-Centric Advancements Chair in AI, Curtin University and Australian National University(Curtin大学人工智能人本发展主席职位,澳大利亚国立大学)
Enhancing the Reliability of Segment Anything Model for Auto-Prompting Medical Image Segmentation with Uncertainty Rectification
Artificial Intelligence Innovation and Incubation Institute, Fudan University(复旦大学人工智能创新与孵化研究院);Shanghai Academy of Artificial Intelligence for Science(上海科学人工智能研究院)
CLIP-AUTT: Test-Time Personalization with Action Unit Prompting for Fine-Grained Video Emotion Recognition
LIVIA, ILLS, Dept. of Systems Engineering, ETS Montreal(LIVIA, ILLS, 系统工程学院, 蒙特利尔高等技术学院);LIVIA, Dept. of Software and IT Engineering, ETS Montreal(LIVIA, 软件与IT工程学院, 蒙特利尔高等技术学院);Dept. of Computer Science, École Polytechnique(计算机科学系, 巴黎综合理工学院)
SenCLIP: Enhancing zero-shot land-use mapping for Sentinel-2 with ground-level prompting
Mediterranean Agronomic Institute of Montpellier - CIHEAM-IAMM(地中海农业研究院-CIHEAM-IAMM);Inria(法国国家信息与自动化技术研究院);INRAE(法国国家农业研究咨询中心);Cirad(国际热带农业研究中心);UMR TETIS(TETIS联合研究单位);Univ. of Montpellier(蒙彼利埃大学)
IF-Bench: Benchmarking and Enhancing MLLMs for Infrared Images with Generative Visual Prompting
MAIS, Institute of Automation(自动化研究所信息处理中心);School of Artificial Intelligence, UCAS(中国科学院大学人工智能学院);Research Center of Aerospace Information, Institute of Automation(自动化研究所航空信息研究中心)
Enhancing Multimodal Emotion Recognition via Multi-Feature Encoding and Attention-Based Fusion
College of Computer Science and Technology, Jilin University(吉林大学计算机科学与技术学院);Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University(吉林大学符号计算与知识工程教育部重点实验室)
Visual Affect Analysis: Predicting Emotions of Image Viewers with Vision-Language Models
Faculty of Mathematics and Computer Science, Adam Mickiewicz University(数学与计算机科学学院,亚当·密茨凯维奇大学);Faculty of Computer Science and Information Technology, Universiti Malaya(计算机科学与信息技术学院,马来亚大学);Department of Computer Science and Engineering, Korea University(计算机科学与工程系,韩国大学);Department of Psychology, University of Texas at Austin(心理学系,德克萨斯大学奥斯汀分校);Cognitive Neuroscience Center, Adam Mickiewicz University(认知神经科学中心,亚当·密茨凯维奇大学)