Human-Object Interaction (HOI) detection is an essential task to understand human-centric images from a fine-grained perspective. Although end-to-end HOI detection models thrive, their paradigm of parallel human/object detection and verb class prediction loses two-stage methods' merit: object-guided hierarchy. The object in one HOI triplet gives direct clues to the verb to be predicted. In this paper, we aim to boost end-to-end models with object-guided statistical priors. Specifically, We propose to utilize a Verb Semantic Model (VSM) and use semantic aggregation to profit from this object-guided hierarchy. Similarity KL (SKL) loss is proposed to optimize VSM to align with the HOI dataset's priors. To overcome the static semantic embedding problem, we propose to generate cross-modality-aware visual and semantic features by Cross-Modal Calibration (CMC). The above modules combined composes Object-guided Cross-modal Calibration Network (OCN). Experiments conducted on two popular HOI detection benchmarks demonstrate the significance of incorporating the statistical prior knowledge and produce state-of-the-art performances. More detailed analysis indicates proposed modules serve as a stronger verb predictor and a more superior method of utilizing prior knowledge. The codes are available at \url{https://github.com/JacobYuan7/OCN-HOI-Benchmark}.
$\mathbf{C}^2$Former: Calibrated and Complementary Transformer for RGB-Infrared Object Detection
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；Institute of Artificial Intelligence, Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院人工智能研究所）
A Calibrated Memorization Index (MI) for Detecting Training Data Leakage in Generative MRI Models
Department of Computer Science, University of York（约克大学计算机科学系）；School of Computer Science, University of Leeds（利兹大学计算机科学学院）；Department of Computing and Mathematics, Manchester Metropolitan University（曼彻斯特 Metropolitan 大学计算与数学系）；Department of Computer Science, University of Manchester（曼彻斯特大学计算机科学系）；Department of Cardiovascular Sciences, KU Leuven（鲁汶大学心血管科学系）
Despite the achievements of large-scale multimodal pre-training approaches, cross-modal retrieval, e.g., image-text retrieval, remains a challenging task. To bridge the semantic gap between the two modalities, previous studies mainly focus on word-region alignment at the object level, lacking the matching between the linguistic relation among the words and the visual relation among the regions. The neglect of such relation consistency impairs the contextualized representation of image-text pairs and hinders the model performance and the interpretability. In this paper, we first propose a novel…
Unleashing the Potential of Consistency Learning for Detecting and Grounding Multi-Modal Media Manipulation
MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所MAIS）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；CAIR, HKISI, Chinese Academy of Sciences（中国科学院计算机辅助研究部）；School of Computer Science and Engineering, the Faculty of Innovation Engineering, M.U.S.T（慕斯科技大学计算机科学与工程学院）；Sangfor Technologies Inc.（Sangfor技术有限公司）；Beijing Jiaotong University（北京交通大学）；Alibaba Group（阿里巴巴集团）
ASAP: Advancing Semantic Alignment Promotes Multi-Modal Manipulation Detecting and Grounding
School of Computer Science and Information Engineering, Hefei University of Technology, China（计算机科学与信息工程学院，合肥工业大学，中国）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, China（人工智能研究所，合肥综合性国家科学中心，中国）
DHCP: Detecting Hallucinations by Cross-modal Attention Pattern in Large Vision-Language Models
Tsinghua University, Tencent（清华大学、腾讯）；Tencent（腾讯）；Tencent, University of Macau（腾讯、澳门大学）；University of Science and Technology Beijing（北京科技大学）；Tsinghua University（清华大学）
In the past few years, there has been a surge of interest in multi-modal problems, from image captioning to visual question answering and beyond. In this paper, we focus on hate speech detection in multi-modal memes wherein memes pose an interesting multi-modal fusion problem. We aim to solve the Facebook Meme Challenge \cite{kiela2020hateful} which aims to solve a binary classification problem of predicting whether a meme is hateful or not. A crucial characteristic of the challenge is that it includes "benign confounders" to counter the possibility of models exploiting unimodal priors. The ch…
Distracted driving is deadly, claiming 3,477 lives in the U.S. in 2015 alone. Although there has been a considerable amount of research on modeling the distracted behavior of drivers under various conditions, accurate automatic detection using multiple modalities and especially the contribution of using the speech modality to improve accuracy has received little attention. This paper introduces a new multimodal dataset for distracted driving behavior and discusses automatic distraction detection using features from three modalities: facial expression, speech and car signals. Detailed multimoda…
Large-scale dissemination of disinformation online intended to mislead or deceive the general population is a major societal problem. Rapid progression in image, video, and natural language generative models has only exacerbated this situation and intensified our need for an effective defense mechanism. While existing approaches have been proposed to defend against neural fake news, they are generally constrained to the very limited setting where articles only have text and metadata such as the title and authors. In this paper, we introduce the more realistic and challenging task of defending …
Nanyang Technological University, Singapore（新加坡南洋理工大学）；DSO National Laboratories, Singapore（新加坡国防科技研究院）
Detecting deception by human behaviors is vital in many fields such as custom security and multimedia anti-fraud. Recently, audio-visual deception detection attracts more attention due to its better performance than using only a single modality. However, in real-world multi-modal settings, the integrity of data can be an issue (e.g., sometimes only partial modalities are available). The missing modality might lead to a decrease in performance, but the model still learns the features of the missed modality. In this paper, to further improve the performance and overcome the missing modality prob…
Unsupervised Multimodal Deepfake Detection Using Intra- and Cross-Modal Inconsistencies
USC Information Sciences Institute（南加州大学信息科学研究所）；USC Ming Hsieh Department of Electrical and Computer Engineering（南加州大学明氏电气与计算机工程学院）；Clemson University（克莱姆森大学）；Holcombe Department of Electrical and Computer Engineering（霍尔科姆电气与计算机工程学院）
This paper attacks the challenging problem of violence detection in videos. Different from existing works focusing on combining multi-modal features, we go one step further by adding and exploiting subclasses visually related to violence. We enrich the MediaEval 2015 violence dataset by \emph{manually} labeling violence videos with respect to the subclasses. Such fine-grained annotations not only help understand what have impeded previous efforts on learning to fuse the multi-modal features, but also enhance the generalization ability of the learned fusion to novel test data. The new subclass …
Prison facilities, mental correctional institutions, sports bars and places of public protest are prone to sudden violence and conflicts. Surveillance systems play an important role in mitigation of hostile behavior and improvement of security by detecting such provocative and aggressive activities. This research proposed using automatic aggressive behavior and anger detection to improve the effectiveness of the surveillance systems. An emotion and aggression aware component will make the surveillance system highly responsive and capable of alerting the security guards in real time. This resea…
Multi-modal user interface control detection using cross-attention
AI Research Lab, Tricentis（Tricentis AI研究实验室）；Institute of Artificial Intelligence, Center for Medical Statistics, Informatics, and Intelligent Systems, Medical University of Vienna（维也纳医科大学人工智能研究所、医学统计、信息学与智能系统中心）
CSIRO-Data61（联邦科学与工业研究组织数据61所）；University of Moratuwa（莫拉图瓦大学）；The University Sydney（悉尼大学）
Counterfeit apps impersonate existing popular apps in attempts to misguide users to install them for various reasons such as collecting personal information or spreading malware. Many counterfeits can be identified once installed, however even a tech-savvy user may struggle to detect them before installation. To this end, this paper proposes to leverage the recent advances in deep learning methods to create image and text embeddings so that counterfeit apps can be efficiently identified when they are submitted for publication. We show that a novel approach of combining content embeddings and s…
AIT Austrian Institute of Technology（奥地利理工学院）；Automation & Control Institute, Technical University of Vienna（维也纳技术大学自动化与控制研究所）；Chair of Information Theory and Data Analytics (INDA), RWTH Aachen University（亚琛工业大学信息理论与数据分析教席）
Adversarial attacks pose a threat to deep learning models. However, research on adversarial detection methods, especially in the multi-modal domain, is very limited. In this work, we propose an efficient and straightforward detection method based on the temporal correlation between audio and video streams. The main idea is that the correlation between audio and video in adversarial examples will be lower than benign examples due to added adversarial noise. We use the synchronisation confidence score as a proxy for audiovisual correlation and based on it we can detect adversarial attacks. To th…
