Situation Recognition is the task of generating a structured summary of what is happening in an image using an activity verb and the semantic roles played by actors and objects. In this task, the same activity verb can describe a diverse set of situations as well as the same actor or object category can play a diverse set of semantic roles depending on the situation depicted in the image. Hence a situation recognition model needs to understand the context of the image and the visual-linguistic meaning of semantic roles. Therefore, we leverage the CLIP foundational model that has learned the context of images via language descriptions. We show that deeper-and-wider multi-layer perceptron (MLP) blocks obtain noteworthy results for the situation recognition task by using CLIP image and text embedding features and it even outperforms the state-of-the-art CoFormer, a Transformer-based model, thanks to the external implicit visual-linguistic knowledge encapsulated by CLIP and the expressive power of modern MLP block designs. Motivated by this, we design a cross-attention-based Transformer using CLIP visual tokens that model the relation between textual roles and visual entities. Our cross-attention-based Transformer known as ClipSitu XTF outperforms existing state-of-the-art by a large margin of 14.1\% on semantic role labelling (value) for top-1 accuracy using imSitu dataset. {Similarly, our ClipSitu XTF obtains state-of-the-art situation localization performance.} We will make the code publicly available.
CoDoL: Conditional Domain Prompt Learning for Out-of-Distribution Generalization
East China Normal University(东华师范大学);Xidian University(西安电子科技大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));The University of Oxford(牛津大学);Tsinghua University(清华大学)
Long-Term TalkingFace Generation via Motion-Prior Conditional Diffusion Model
Nanjing University of Science and Technology(南京理工大学);Nanjing University(南京大学);Tongji University(同济大学);Peking University(北京大学);Sun Yat-sen University(中山大学);National University of Singapore(新加坡国立大学)
RATLIP: Generative Adversarial CLIP Text-to-Image Synthesis Based on Recurrent Affine Transformations
School of Artificial Intelligence, Guangxi Colleges and Universities Key Laboratory of AI Algorithm Engineering(人工智能学院、广西 Colleges and Universities AI 算法工程重点实验室)
DeltaEdit: Exploring Text-free Training for Text-Driven Image Manipulation
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);CRIPAC, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所CRIPAC);VIS, Baidu Inc.(百度公司VIS)
Accelerating Conditional Prompt Learning via Masked Image Modeling for Vision-Language Models
Convergence Research Institute, Sungkyunkwan University(convergence research institute, 首尔大学);School of Information Technology, Deakin University(信息科技学院, 德金大学);Department of Electrical and Computer Engineering, Sungkyunkwan University(电气与计算机工程系, 首尔大学)
GMGaze: MoE-Based Context-Aware Gaze Estimation with CLIP and Multiscale Transformer
School of Artificial Intelligence, Guilin University of Electronic Technology, Guilin, China(桂林电子科技大学人工智能学院);The Laboratory for Imagery, Vision and Artificial Intelligence, École de Technologie Supérieure, Montreal, Canada(图像、视觉与人工智能实验室,蒙特利尔大学高级技术学院);Information Technology Department, College of Computer and Information Sciences, King Saud University, Riyadh, Saudi Arabia(计算机与信息科学学院信息科技系,沙特国王大学);Department of Cybersecurity, College of Computer Science and Engineering, Taibah University, Medina, Saudi Arabia(计算机科学与工程学院网络安全系,塔伊卜大学);Department of Networked Engineering, College of Computer Science and Engineering, Jeddah University, Jeddah, Saudi Arabia(计算机科学与工程学院网络工程系,吉达大学)
Peer is Your Pillar: A Data-unbalanced Conditional GANs for Few-shot Image Generation
University of Science and Technology of China(中国科学技术大学);The University of Sydney(悉尼大学);JD Explore Academic(京东探索研究院);Chongqing University of Posts and Telecommunications(重庆邮电大学);CAS Key Laboratory of Technology in Geo-spatial Information Processing and Application System(中国科学院地理空间信息处理与应用系统技术重点实验室)
GazeFormer-MoE: Context-Aware Gaze Estimation via CLIP and MoE Transformer
AIPM, School of Artificial Intelligence, Guilin University of Electronic Technology, China(AIPM,人工智能学院,桂林电子科技大学,中国);Laboratory for Imagery, Vision and Artificial Intelligence, École de Technologie Supérieure, Canada(影像、视觉与人工智能实验室,École de Technologie Supérieure,加拿大)
ICPL-ReID: Identity-Conditional Prompt Learning for Multi-Spectral Object Re-Identification
Information Materials and Intelligent Sensing Laboratory of Anhui Province(安徽省信息材料与智能感知实验室);Anhui Provincial Key Laboratory of Security Artificial Intelligence(安徽省安全人工智能重点实验室);School of Artificial Intelligence, Anhui University(安徽大学人工智能学院)
Alpha-CLIP: A CLIP Model Focusing on Wherever You Want
Shanghai AI Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);University of Macau(澳门大学);MThreads, Inc.(摩尔线程公司);Shanghai Jiao Tong University(上海交通大学);Fudan University(复旦大学)
Purdue University(普渡大学);Etsy, Inc(Etsy公司);University at Albany, State University of New York(纽约州立大学奥尔巴尼分校);University of South Australia(南澳大学);Sapienza University of Rome(罗马大学)
Unified Coding for Both Human Perception and Generalized Machine Analytics with CLIP Supervision
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东省人工智能与数字经济实验室(深圳));Peng Cheng Laboratory(鹏城实验室);Shenzhen University(深圳大学);City University of Hong Kong(香港城市大学)