Exploring Conditional Multi-Modal Prompts for Zero-shot HOI Detection
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）
Zero-shot Human-Object Interaction (HOI) detection has emerged as a frontier topic due to its capability to detect HOIs beyond a predefined set of categories. This task entails not only identifying the interactiveness of human-object pairs and localizing them but also recognizing both seen and unseen interaction categories. In this paper, we introduce a novel framework for zero-shot HOI detection using Conditional Multi-Modal Prompts, namely CMMP. This approach enhances the generalization of large foundation models, such as CLIP, when fine-tuned for HOI detection. Unlike traditional prompt-learning methods, we propose learning decoupled vision and language prompts for interactiveness-aware visual feature extraction and generalizable interaction classification, respectively. Specifically, we integrate prior knowledge of different granularity into conditional vision prompts, including an input-conditioned instance prior and a global spatial pattern prior. The former encourages the image encoder to treat instances belonging to seen or potentially unseen HOI concepts equally while the latter provides representative plausible spatial configuration of the human and object under interaction. Besides, we employ language-aware prompt learning with a consistency constraint to preserve the knowledge of the large foundation model to enable better generalization in the text branch. Extensive experiments demonstrate the efficacy of our detector with conditional multi-modal prompts, outperforming previous state-of-the-art on unseen classes of various zero-shot settings. The code and models are available at \url{https://github.com/ltttpku/CMMP}.
Zhejiang University（浙江大学）；State Key Laboratory of Industrial Control Technology, Zhejiang University（浙江大学工业控制技术国家重点实验室）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）
Human-Object Interaction (HOI) detection is an essential task to understand human-centric images from a fine-grained perspective. Although end-to-end HOI detection models thrive, their paradigm of parallel human/object detection and verb class prediction loses two-stage methods' merit: object-guided hierarchy. The object in one HOI triplet gives direct clues to the verb to be predicted. In this paper, we aim to boost end-to-end models with object-guided statistical priors. Specifically, We propose to utilize a Verb Semantic Model (VSM) and use semantic aggregation to profit from this object-gu…
Uni-HOI:A Unified framework for Learning the Joint distribution of Text and Human-Object Interaction
Wuhan University（武汉大学）；Peking University（北京大学）；Institute for Clarity in Documentation（文档清晰研究所）；Inria Paris-Rocquencourt（巴黎-罗quentcourt研究所）；Rajiv Gandhi University（拉贾·甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕勒研究实验室）
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；JD Explore Academy（京东探索研究院）；The Hong Kong Polytechnic University（香港理工大学）
Recent high-performing Human-Object Interaction (HOI) detection techniques have been highly influenced by Transformer-based object detector (i.e., DETR). Nevertheless, most of them directly map parametric interaction queries into a set of HOI predictions through vanilla Transformer in a one-stage manner. This leaves rich inter- or intra-interaction structure under-exploited. In this work, we design a novel Transformer-style HOI detector, i.e., Structure-aware Transformer over Interaction Proposals (STIP), for HOI detection. Such design decomposes the process of HOI set prediction into two subs…
Generative Human-Object Interaction Detection via Differentiable Cognitive Steering of Multi-modal LLMs
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University（上海交通大学图像通信与网络工程研究所）；Xi’an Jiao Tong University（西安交通大学）；Shandong University（山东大学）；Tianjin University（天津大学）
Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibration
Wangxuan Institute of Computer Technology, Peking University（北京大学王璇计算机技术研究所）；National Institute of Health Data Science, Peking University（北京大学国家健康数据科学研究院）
Multimodal Priors-Augmented Text-Driven 3D Human-Object Interaction Generation
State Key Laboratory of Virtual Reality Technology and Systems（虚拟现实技术与系统国家重点实验室）；Beihang University（北京航空航天大学）；Department of Computer Science, University of Durham（杜伦大学计算机科学系）；Zhongguancun Laboratory（中关村实验室）
Human-Object Interaction (HOI) detection devotes to learn how humans interact with surrounding objects. Latest end-to-end HOI detectors are short of relation reasoning, which leads to inability to learn HOI-specific interactive semantics for predictions. In this paper, we therefore propose novel relation reasoning for HOI detection. We first present a progressive Relation-aware Frame, which brings a new structure and parameter sharing pattern for interaction inference. Upon the frame, an Interaction Intensifier Module and a Correlation Parsing Module are carefully designed, where: a) interacti…
School of Electronic and Information Engineering, South China University of Technology（华南理工大学电子与信息学院）；Pazhou Lab（琶洲实验室）；JD Explore Academy at JD.com（京东探索研究院）
Human-Object Interaction (HOI) detection is important to human-centric scene understanding tasks. Existing works tend to assume that the same verb has similar visual characteristics in different HOI categories, an approach that ignores the diverse semantic meanings of the verb. To address this issue, in this paper, we propose a novel Polysemy Deciphering Network (PD-Net) that decodes the visual polysemy of verbs for HOI detection in three distinct ways. First, we refine features for HOI detection to be polysemyaware through the use of two novel modules: namely, Language Prior-guided Channel At…
SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarios
School of Software Engineering, South China University of Technology（华南理工大学软件工程学院）；Department of Automation, Tsinghua University（清华大学自动化系）；School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）
HOIGaze: Gaze Estimation During Hand-Object Interactions in Extended Reality Exploiting Eye-Hand-Head Coordination
University of Stuttgart Germany（斯图加特大学）；University of Tuebingen Germany（图宾根大学）；The Center for Bionic Intelligence Tuebingen Stuttgart Germany（图宾根-斯图加特生物智能中心）
I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactions
ShanghaiTech University（上海科技大学）；Shanghai Advanced Research Institute, Chinese Academy of Sciences（中国科学院上海高等研究院）；University of Chinese Academy of Sciences（中国科学院大学）
Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics（西南财经大学计算机与人工智能学院）；Nanjing Forestry University（南京林业大学）
Human-Object Interaction (HOI) detection lies at the core of action understanding. Besides 2D information such as human/object appearance and locations, 3D pose is also usually utilized in HOI learning since its view-independence. However, rough 3D body joints just carry sparse body information and are not sufficient to understand complex interactions. Thus, we need detailed 3D body shape to go further. Meanwhile, the interacted object in 3D is also not fully studied in HOI learning. In light of these, we propose a detailed 2D-3D joint representation learning method. First, we utilize the sing…
Mitigating Long-Tail Bias in HOI Detection via Adaptive Diversity Cache
College of Future Information Technology, Fudan University（复旦大学未来信息技术学院）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学与技术学院）；Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究院（TeleAI））
University of Maryland, College Park（马里兰大学帕克分校）；Honda Research Institute, USA（本田美国研究所）
We cast visual retrieval as a regression problem by posing triplet loss as a regression loss. This enables epistemic uncertainty estimation using dropout as a Bayesian approximation framework in retrieval. Accordingly, Monte Carlo (MC) sampling is leveraged to boost retrieval performance. Our approach is evaluated on two applications: person re-identification and autonomous car driving. Comparable state-of-the-art results are achieved on multiple datasets for the former application. We leverage the Honda driving dataset (HDD) for autonomous car driving application. It provides multiple modalit…
Exploring Multi-Modal Contextual Knowledge for Open-Vocabulary Object Detection
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of the Chinese Academy of Sciences（中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）；Tencent Youtu Lab（腾讯优图实验室）
University of Science and Technology of China（中国科学技术大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
The audio-video based emotion recognition aims to classify a given video into basic emotions. In this paper, we describe our approaches in EmotiW 2019, which mainly explores emotion features and feature fusion strategies for audio and visual modality. For emotion features, we explore audio feature with both speech-spectrogram and Log Mel-spectrogram and evaluate several facial features with different CNN models and different emotion pretrained strategies. For fusion strategies, we explore intra-modal and cross-modal fusion methods, such as designing attention mechanisms to highlights important…
AVS-Mamba: Exploring Temporal and Multi-modal Mamba for Audio-Visual Segmentation
School of Information and Communication Engineering, Dalian University of Technology（大连理工大学信息与通信工程学院）；School of Innovation and Entrepreneurship, Dalian University of Technology（大连理工大学创新创业学院）；School of Artificial Intelligence, Dalian University of Technology（大连理工大学人工智能学院）
TriDi: Trilateral Diffusion of 3D Humans, Objects, and Interactions
University of Tübingen, Germany（图宾根大学）；Tübingen AI Center, Germany（图宾根人工智能中心）；Technical University of Munich, Germany（慕尼黑技术大学）；Munich Center for Machine Learning, Germany（慕尼黑机器学习中心）；Max Planck Institute for Informatics, Saarland Informatics Campus, Germany（马克斯·普朗克信息研究所）
It is counter-intuitive that multi-modality methods based on point cloud and images perform only marginally better or sometimes worse than approaches that solely use point cloud. This paper investigates the reason behind this phenomenon. Due to the fact that multi-modality data augmentation must maintain consistency between point cloud and images, recent methods in this field typically use relatively insufficient data augmentation. This shortage makes their performance under expectation. Therefore, we contribute a pipeline, named transformation flow, to bridge the gap between single and multi-…
Peking University（北京大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
Recently, attention-based encoder-decoder models have been used extensively in image captioning. Yet there is still great difficulty for the current methods to achieve deep image understanding. In this work, we argue that such understanding requires visual attention to correlated image regions and semantic attention to coherent attributes of interest. Based on the Transformer, to perform effective attention, we explore image captioning from a cross-modal perspective and propose the Global-and-Local Information Exploring-and-Distilling approach that explores and distills the source information …
Institute for Datability Science（数据可持续性科学研究所）；Osaka University（大阪大学）
In computer vision, visual arts are often studied from a purely aesthetics perspective, mostly by analysing the visual appearance of an artistic reproduction to infer its style, its author, or its representative features. In this work, however, we explore art from both a visual and a language perspective. Our aim is to bridge the gap between the visual appearance of an artwork and its underlying meaning, by jointly analysing its aesthetics and its semantics. We introduce the use of multi-modal techniques in the field of automatic art analysis by 1) collecting a multi-modal dataset with fine-ar…
Exploring Multi-Modal Control in Music-Driven Dance Generation
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Peng Cheng Laboratory（鹏城实验室）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）
University of Chinese Academy of Sciences（中国科学院大学）；Monash University（莫纳什大学）；Sun Yat-sen University（中山大学）；Dark Matter AI Inc.（暗物质人工智能公司）
Vision-dialog navigation posed as a new holy-grail task in vision-language disciplinary targets at learning an agent endowed with the capability of constant conversation for help with natural language and navigating according to human responses. Besides the common challenges faced in visual language navigation, vision-dialog navigation also requires to handle well with the language intentions of a series of questions about the temporal context from dialogue history and co-reasoning both dialogs and visual scenes. In this paper, we propose the Cross-modal Memory Network (CMN) for remembering an…
In autonomous driving, LiDAR point-clouds and RGB images are two major data modalities with complementary cues for 3D object detection. However, it is quite difficult to sufficiently use them, due to large inter-modal discrepancies. To address this issue, we propose a novel framework, namely Contrastively Augmented Transformer for multi-modal 3D object Detection (CAT-Det). Specifically, CAT-Det adopts a two-stream structure consisting of a Pointformer (PT) branch, an Imageformer (IT) branch along with a Cross-Modal Transformer (CMT) module. PT, IT and CMT jointly encode intra-modal and inter-m…
