Creative works, whether paintings or memes, follow unique journeys that result in their final form. Understanding these journeys, a process known as "provenance analysis", provides rich insights into the use, motivation, and authenticity underlying any given work. The application of this type of study to the expanse of unregulated content on the Internet is what we consider in this paper. Provenance analysis provides a snapshot of the chronology and validity of content as it is uploaded, re-uploaded, and modified over time. Although still in its infancy, automated provenance analysis for online multimedia is already being applied to different types of content. Most current works seek to build provenance graphs based on the shared content between images or videos. This can be a computationally expensive task, especially when considering the vast influx of content that the Internet sees every day. Utilizing non-content-based information, such as timestamps, geotags, and camera IDs can help provide important insights into the path a particular image or video has traveled during its time on the Internet without large computational overhead. This paper tests the scope and applicability of metadata-based inferences for provenance graph construction in two different scenarios: digital image forensics and cultural analytics.
We propose a novel method for predicting image labels by fusing image content descriptors with the social media context of each image. An image uploaded to a social media site such as Flickr often has meaningful, associated information, such as comments and other images the user has uploaded, that is complementary to pixel content and helpful in predicting labels. Prediction challenges such as ImageNet~\cite{imagenet_cvpr09} and MSCOCO~\cite{LinMBHPRDZ:ECCV14} use only pixels, while other methods make predictions purely from social media context \cite{McAuleyECCV12}. Our method is based on a n…
PRIMA: Pre-Training with Risk-Integrated Image--Metadata Alignment for Medical Diagnosis with LLM-Based Feature Aggregation
Department of Biomedical Engineering, Duke University（杜克大学生物医学工程系）；Department of Ophthalmology and Visual Sciences, University of Michigan（密歇根大学眼科与视觉科学系）
Indian Institute of Science（印度科学学院）；Ola Electric（奥拉电气）；Verisk AI Research（韦里斯克人工智能研究院）
Recent algorithms for image manipulation detection almost exclusively use deep network models. These approaches require either dense pixelwise groundtruth masks, camera ids, or image metadata to train the networks. On one hand, constructing a training set to represent the countless tampering possibilities is impractical. On the other hand, social media platforms or commercial applications are often constrained to remove camera ids as well as metadata from images. A self-supervised algorithm for training manipulation detection models without dense groundtruth or camera/image metadata would be e…
Nanyang Technological University（南洋理工大学）；Peng Cheng Lab（鹏城实验室）；The Hong Kong Polytechnic University（香港理工大学）
While raw images exhibit advantages over sRGB images (e.g., linearity and fine-grained quantization level), they are not widely used by common users due to the large storage requirements. Very recent works propose to compress raw images by designing the sampling masks in the raw image pixel space, leading to suboptimal image representations and redundant metadata. In this paper, we propose a novel framework to learn a compact representation in the latent space serving as the metadata in an end-to-end manner. Furthermore, we propose a novel sRGB-guided context model with improved entropy estima…
Anveshak - A Groundtruth Generation Tool for Foreground Regions of Document Images
Department of Computer Science and Engineering, Indian Institute of Technology Kharagpur（计算机科学与工程系，印度理工学院卡拉克普尔分校）；Indian Institute of Technology Kharagpur（印度理工学院卡拉克普尔分校）；Department of Computer Science and Engineering（计算机科学与工程系）
York University（约克大学）；Samsung AI Center – Toronto（三星人工智能中心多伦多分部）
Most camera images are rendered and saved in the standard RGB (sRGB) format by the camera's hardware. Due to the in-camera photo-finishing routines, nonlinear sRGB images are undesirable for computer vision tasks that assume a direct relationship between pixel values and scene radiance. For such applications, linear raw-RGB sensor images are preferred. Saving images in their raw-RGB format is still uncommon due to the large storage requirement and lack of support by many imaging applications. Several "raw reconstruction" methods have been proposed that utilize specialized metadata sampled from…
Purdue University（普渡大学）；Politecnico di Milano（米兰理工大学）
Manipulating video content is easier than ever. Due to the misuse potential of manipulated content, multiple detection techniques that analyze the pixel data from the videos have been proposed. However, clever manipulators should also carefully forge the metadata and auxiliary header information, which is harder to do for videos than images. In this paper, we propose to identify forged videos by analyzing their multimedia stream descriptors with simple binary classifiers, completely avoiding the pixel space. Using well-known datasets, our results show that this scalable approach can achieve a …
University of Malta（马耳他大学）；University of Edinburgh（爱丁堡大学）
Convolutional Neural Networks (CNNs) have achieved impressive results across many super-resolution (SR) and image restoration tasks. While many such networks can upscale low-resolution (LR) images using just the raw pixel-level information, the ill-posed nature of SR can make it difficult to accurately super-resolve an image which has undergone multiple different degradations. Additional information (metadata) describing the degradation process (such as the blur kernel applied, compression level, etc.) can guide networks to super-resolve LR images with higher fidelity to the original source. P…
GraphDerm: Fusing Imaging, Physical Scale, and Metadata in a Population-Graph Classifier for Dermoscopic Lesions
School of Computer Science, Institute for Research in Fundamental Sciences (IPM)（计算机科学学院，基础科学研究所（IPM））；Department of Physics, Shahid Beheshti University（物理系，肖德·贝赫谢什提大学）；Faculty of Science, University of Tehran（科学学院，德黑兰大学）；Fatemeh Zahra Hospital, Alborz University of Medical Sciences（法特梅·扎拉医院，阿尔bor兹医学科学大学）；Department of Electrical and Computer Engineering, Sungkyunkwan University（电气与计算机工程系，成均馆大学）；Department of Computer Science and Engineering, Shahid Beheshti University（计算机科学与工程系，肖德·贝赫谢什提大学）；Autoimmune Bullous Diseases Research Center, Razi Hospital, Tehran University of Medical Sciences（自身免疫性水疱病研究中心，拉齐医院，德黑兰医学科学大学）
