A dramatic rise in the flow of manipulated image content on the Internet has led to an aggressive response from the media forensics research community. New efforts have incorporated increased usage of techniques from computer vision and machine learning to detect and profile the space of image manipulations. This paper addresses Image Provenance Analysis, which aims at discovering relationships among different manipulated image versions that share content. One of the main sub-problems for provenance analysis that has not yet been addressed directly is the edit ordering of images that share full content or are near-duplicates. The existing large networks that generate image descriptors for tasks such as object recognition may not encode the subtle differences between these image covariates. This paper introduces a novel deep learning-based approach to provide a plausible ordering to images that have been generated from a single image through transformations. Our approach learns transformation-aware descriptors using weak supervision via composited transformations and a rank-based quadruplet loss. To establish the efficacy of the proposed approach, comparisons with state-of-the-art handcrafted and deep learning-based descriptors, and image matching approaches are made. Further experimentation validates the proposed approach in the context of image provenance analysis.
Embedding Aggregation for Forensic Facial Comparison
University of Brasilia（巴西利亚大学）；National Institute of Criminalistics（国家刑事科学研究所）；University of Beira Interior（贝拉内瓦大学）；Netherlands Forensic Institute（荷兰法医研究所）
Fake videos represent an important misinformation threat. While existing forensic networks have demonstrated strong performance on image forgeries, recent results reported on the Adobe VideoSham dataset show that these networks fail to identify fake content in videos. In this paper, we show that this is due to video coding, which introduces local variation into forensic traces. In response, we propose VideoFACT - a new network that is able to detect and localize a wide variety of video forgeries and manipulations. To overcome challenges that existing networks face when analyzing videos, our ne…
University Federico II of Naples（那不勒斯费德里科二世大学）；Technical University of Munich（慕尼黑工业大学）；University of Erlangen-Nuremberg（埃尔朗根-纽伦堡大学）
Distinguishing manipulated from real images is becoming increasingly difficult as new sophisticated image forgery approaches come out by the day. Naive classification approaches based on Convolutional Neural Networks (CNNs) show excellent performance in detecting image manipulations when they are trained on a specific forgery method. However, on examples from unseen manipulation approaches, their performance drops significantly. To address this limitation in transferability, we introduce Forensic-Transfer (FT). We devise a learning-based forensic detector which adapts well to new domains, i.e.…
Flow of Truth: Proactive Temporal Forensics for Image-to-Video Generation
Anhui Province Key Laboratory of Digital Security (School of Cyber Science and Technology, University of Science and Technology of China)（安徽省数字安全重点实验室（网络安全学院，中国科学技术大学））
Bridging the Modality Gap in Forensic Image Retrieval
Advanced Technologies Application Center (CENATAV)（先进技术应用中心（CENATAV））；Centro de Sistemas Complejos, Facultad de Física, Universidad de La Habana（哈瓦那大学物理学院复杂系统中心）
Tell-Tale Watermarks for Explanatory Reasoning in Synthetic Media Forensics
Information and Society Research Division, National Institute of Informatics（信息与社会研究部，国家信息研究所）；Graduate School of Information Science and Technology, University of Tokyo（东京大学信息科学与技术研究生院）
Inconsistency in contrast enhancement can be used to expose image forgeries. In this work, we describe a new method to estimate contrast enhancement from a single image. Our method takes advantage of the nature of contrast enhancement as a mapping between pixel values, and the distinct characteristics it introduces to the image pixel histogram. Our method recovers the original pixel histogram and the contrast enhancement simultaneously from a single image with an iterative algorithm. Unlike previous methods, our method is robust in the presence of additive noise perturbations that are used to …
Propose and Rectify: A Forensics-Driven MLLM Framework for Image Manipulation Localization
Department of Electrical Engineering, City University of Hong Kong（香港城市大学电子工程系）；Rapid-Rich Object Search (ROSE) Lab, School of Electrical and Electronic Engineering, Nanyang Technology University（南洋理工大学电子与电气工程学院快速丰富对象搜索（ROSE）实验室）；Shanghai Jiao Tong University（上海交通大学）
One common task in image forensics is to detect spliced images, where multiple source images are composed to one output image. Most of the currently best performing splicing detectors leverage high-frequency artifacts. However, after an image underwent strong compression, most of the high frequency artifacts are not available anymore. In this work, we explore an alternative approach to splicing detection, which is potentially better suited for images in-the-wild, subject to strong compression and downsampling. Our proposal is to model the color formation of an image. The color formation largel…
VAAS: Vision-Attention Anomaly Scoring for Image Manipulation Detection in Digital Forensics
Forensics and Security Research Group, South East Technological University（法医与安全研究组，南东技术大学）；Security Research Group, School of Computer Science, University College Dublin（安全研究组，计算机科学学院，都柏林大学学院）
In this paper, we develop position embedding transformation (PETR) for multi-view 3D object detection. PETR encodes the position information of 3D coordinates into image features, producing the 3D position-aware features. Object query can perceive the 3D position-aware features and perform end-to-end object detection. PETR achieves state-of-the-art performance (50.4% NDS and 44.1% mAP) on standard nuScenes dataset and ranks 1st place on the benchmark. It can serve as a simple yet strong baseline for future research. Code is available at \url{https://github.com/megvii-research/PETR}.
SAME++: A Self-supervised Anatomical eMbeddings Enhanced medical image registration framework using stable sampling and regularized transformation
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Alibaba Group（阿里巴巴集团）；Johns Hopkins University（约翰斯·霍普金斯大学）；The First Affiliated Hospital of Zhejiang University（浙江大学附属第一医院）
University of Florida（佛罗里达大学）；National Renewable Energy Lab（美国国家可再生能源实验室）
We present a new approach to unsupervised shape correspondence learning between pairs of point clouds. We make the first attempt to adapt the classical locally linear embedding algorithm (LLE) -- originally designed for nonlinear dimensionality reduction -- for shape correspondence. The key idea is to find dense correspondences between shapes by first obtaining high-dimensional neighborhood-preserving embeddings of low-dimensional point clouds and subsequently aligning the source and target embeddings using locally linear transformations. We demonstrate that learning the embedding using a new …
Photometric transformations, such as brightness and contrast adjustment, can be applied to a face image repeatedly creating a set of near-duplicate images. Identifying the original image from a set of such near-duplicates and deducing the relationship between them are important in the context of digital image forensics. This is commonly done by generating an image phylogeny tree \textemdash \hspace{0.08cm} a hierarchical structure depicting the relationship between a set of near-duplicate images. In this work, we utilize three different families of basis functions to model pairwise relationshi…
Registration of point clouds related by rigid transformations is one of the fundamental problems in computer vision. However, a solution to the practical scenario of aligning sparsely and differently sampled observations in the presence of noise is still lacking. We approach registration in this scenario with a fusion of the closed-form Universal Mani-fold Embedding (UME) method and a deep neural network. The two are combined into a single unified framework, named DeepUME, trained end-to-end and in an unsupervised manner. To successfully provide a global solution in the presence of large trans…
Semantics-Oriented Multitask Learning for DeepFake Detection: A Joint Embedding Approach
Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）；Lee Kong Chian School of Medicine, Nanyang Technological University（南洋理工大学李光耀医学院）；Yunnan United Vision Technology Co., Ltd（云南联合视觉科技有限公司）；Department of Computer Science and Engineering, University at Buffalo, State University of New York（纽约州立大学布法罗分校计算机科学与工程系）
MICC – Media Integration and Communication Center（媒体集成与通信中心）；University of Florence（佛罗伦萨大学）
Neural networks are widely used as a model for classification in a large variety of tasks. Typically, a learnable transformation (i.e. the classifier) is placed at the end of such models returning a value for each class used for classification. This transformation plays an important role in determining how the generated features change during the learning process. In this work we argue that this transformation not only can be fixed (i.e. set as non trainable) with no loss of accuracy, but it can also be used to learn stationary and maximally discriminative embeddings. We show that the stationa…
We propose a new embedding method, named Quantile-Quantile Embedding (QQE), for distribution transformation and manifold embedding with the ability to choose the embedding distribution. QQE, which uses the concept of quantile-quantile plot from visual statistical tests, can transform the distribution of data to any theoretical desired distribution or empirical reference sample. Moreover, QQE gives the user a choice of embedding distribution in embedding the manifold of data into the low dimensional embedding space. It can also be used for modifying the embedding distribution of other dimension…
Shahjalal University of Science and Technology（沙赫贾拉尔科技大学）；University of Alberta（阿尔伯塔大学）；Fordham University（福特汉姆大学）
Forensic analysis of manipulated pixels requires the identification of various hidden and subtle features from images. Conventional image recognition models generally fail at this task because they are biased and more attentive toward the dominant local and spatial features. In this paper, we propose a novel Gated Context Attention Network (GCA-Net) that utilizes non-local attention in conjunction with a gating mechanism in order to capture the finer image discrepancies and better identify forged regions. The proposed framework uses high dimensional embeddings to filter and aggregate the relev…
Department of Applied Tumor Biology, Institute of Pathology, Heidelberg University Hospital（海德堡大学医院病理学研究所应用肿瘤生物学系）；National Center for Hereditary Tumor Syndromes, University Hospital Bonn（波恩大学医院遗传性肿瘤综合征国家中心）；Department of Internal Medicine I, University Hospital Bonn（波恩大学医院第一内科）；Leibniz Institut für Wissensmedien（莱布尼茨知识媒体研究所）
Latent text representations exhibit geometric regularities, such as the famous analogy: queen is to king what woman is to man. Such structured semantic relations were not demonstrated on image representations. Recent works aiming at bridging this semantic gap embed images and text into a multimodal space, enabling the transfer of text-defined transformations to the image modality. We introduce the SIMAT dataset to evaluate the task of Image Retrieval with Multimodal queries. SIMAT contains 6k images and 18k textual transformation queries that aim at either replacing scene elements or changing …
The complexity of a learning task is increased by transformations in the input space that preserve class identity. Visual object recognition for example is affected by changes in viewpoint, scale, illumination or planar transformations. While drastically altering the visual appearance, these changes are orthogonal to recognition and should not be reflected in the representation or feature encoding used for learning. We introduce a framework for weakly supervised learning of image embeddings that are robust to transformations and selective to the class distribution, using sets of transforming e…
Universitá di Siena（锡耶纳大学）；Politecnico di Milano（米兰理工大学）
Due to the reduction of technological costs and the increase of satellites launches, satellite images are becoming more popular and easier to obtain. Besides serving benevolent purposes, satellite data can also be used for malicious reasons such as misinformation. As a matter of fact, satellite images can be easily manipulated relying on general image editing tools. Moreover, with the surge of Deep Neural Networks (DNNs) that can generate realistic synthetic imagery belonging to various domains, additional threats related to the diffusion of synthetically generated satellite images are emergin…
New York University（纽约大学）；Viasat, Inc.（Viasat公司）；Facebook AI Research（脸书人工智能研究院）
We present Transformation Invariance and Covariance Contrast (TiCo) for self-supervised visual representation learning. Similar to other recent self-supervised learning methods, our method is based on maximizing the agreement among embeddings of different distorted versions of the same image, which pushes the encoder to produce transformation invariant representations. To avoid the trivial solution where the encoder generates constant vectors, we regularize the covariance matrix of the embeddings from different images by penalizing low rank solutions. By jointly minimizing the transformation i…
Convolutional Neural Networks (CNN) has been widely applied in the realm of computer vision. However, given the fact that CNN models are translation invariant, they are not aware of the coordinate information of each pixel. Thus the generalization ability of CNN will be limited since the coordinate information is crucial for a model to learn affine transformations which directly operate on the coordinate of each pixel. In this project, we proposed a simple approach to incorporate the coordinate information to the CNN model through coordinate embedding. Our approach does not change the downstre…
