This bachelor's thesis was the foundation of the paper "DisguisOR: Holistic Face Anonymization for the Operating Room" (see arXiv:2307.14241), published at IPCAI'23
We propose a novel method that leverages 3D information to automatically anonymize multi-view RGB-D video recordings of operating rooms (OR). Our anonymization method preserves the original data distribution by replacing the faces in each image with different faces so that the data remains suitable for further downstream tasks. In contrast to established anonymization methods, our approach localizes faces in 3D space first rather than in 2D space. Each face is then anonymized by reprojecting a different face back into each camera view, ultimately replacing the original faces in the resulting images. Furthermore, we introduce a multi-view RGB-D dataset, captured during a real operation of experienced surgeons performing laparoscopic surgery on an animal object (swine), which encapsulates typical characteristics of ORs. Finally, we present experimental results evaluated on that dataset, showing that leveraging 3D data can achieve better face localization in OR images and generate more realistic faces than the current state-of-the-art. There has been, to our knowledge, no prior work that addresses the anonymization of multi-view OR recordings, nor 2D face localization that leverages 3D information.
Self-Supervised Uncalibrated Multi-View Video Anonymization in the Operating Room
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, France（斯特拉斯堡大学，法国国家科学研究中心，法国国家卫生研究院，ICube，UMR7357，法国）；University Hospital of Strasbourg, 67000 Strasbourg, France（斯特拉斯堡大学医院，法国）
CRIPAC & NLPR, CASIA（中国科学院自动化研究所模式识别国家重点实验室与智能感知与计算研究中心）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
De-identification of face data has drawn increasing attention in recent years. It is important to protect people's identities meanwhile keeping the utility of the data in many computer vision tasks. We propose a Controllable Face Anonymization Network (CFA-Net), a novel approach that can anonymize the identity of given faces in images and videos, based on a generator that can disentangle face identity from other image contents. We reach the goal of controllable face anonymization through manipulating identity vectors in the generator's identity representation space. Various anonymized faces de…
Norwegian University of Science and Technology（挪威科技大学）
Generative Adversarial Networks (GANs) are widely adapted for anonymization of human figures. However, current state-of-the-art limit anonymization to the task of face anonymization. In this paper, we propose a novel anonymization framework (DeepPrivacy2) for realistic anonymization of human figures and faces. We introduce a new large and diverse dataset for human figure synthesis, which significantly improves image quality and diversity of generated images. Furthermore, we propose a style-based GAN that produces high quality, diverse and editable anonymizations. We demonstrate that our full-b…
iFADIT: Invertible Face Anonymization via Disentangled Identity Transform
Chongqing Key Laboratory of Image Cognition（重庆图像认知重点实验室）；State Key Laboratory of Integrated Services Networks（综合业务网国家重点实验室）；Chongqing University of Posts and Telecommunications（重庆邮电大学）；Xidian University（西安电子科技大学）
Eindhoven University of Technology（埃因霍温理工大学）；ViNotion B.V.（ViNotion公司）；University of Amsterdam（阿姆斯特丹大学）
Since the introduction of the GDPR and CCPA legislation, both public and private facial image datasets are increasingly scrutinized. Several datasets have been taken offline completely and some have been anonymized. However, it is unclear how anonymization impacts face detection performance. To our knowledge, this paper presents the first empirical study on the effect of image anonymization on supervised training of face detectors. We compare conventional face anonymizers with three state-of-the-art Generative Adversarial Network-based (GAN) methods, by training an off-the-shelf face detector …
Local Features Meet Stochastic Anonymization: Revolutionizing Privacy-Preserving Face Recognition for Black-Box Models
National Engineering Research Center for Multimedia Software（国家多媒体软件工程技术研究中心）；Institute of Artificial Intelligence（人工智能研究院）；School of Computer Science（计算机学院）；Wuhan University（武汉大学）；Harbin Institute of Technology（哈尔滨工业大学）
G2Face: High-Fidelity Reversible Face Anonymization via Generative and Geometric Priors
South China University of Technology（华南理工大学）；Guangdong Engineering Center for Large Model and GenAI Technology（广东省大模型与生成式人工智能技术工程中心）；State Key Laboratory of Subtropical Building and Urban Science（亚热带建筑与城市科学国家重点实验室）；The Hong Kong Polytechnic University（香港理工大学）
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；FZI Research Center for Information Technology（FZI信息技术研究中心）
In order to protect vulnerable road users (VRUs), such as pedestrians or cyclists, it is essential that intelligent transportation systems (ITS) accurately identify them. Therefore, datasets used to train perception models of ITS must contain a significant number of vulnerable road users. However, data protection regulations require that individuals are anonymized in such datasets. In this work, we introduce a novel deep learning-based pipeline for face anonymization in the context of ITS. In contrast to related methods, we do not use generative adversarial networks (GANs) but build upon recen…
Secure and reversible face anonymization with diffusion models
LIRMM, Univ Montpellier, CNRS, Montpellier, FRANCE（LIRMM，蒙彼利埃大学，法国国家科学研究中心，蒙彼利埃，法国）；IMT Nord Europe, Institut Mines-Télécom, Univ. Lille, Centre for Digital Systems, Lille, FRANCE（IMT北欧，法国电信研究院，里尔大学，数字系统研究中心，里尔，法国）
Contextual Robotics Institute（情境机器人研究所）；University of California, San Diego（加利福尼亚大学圣迭戈分校）
Many outdoor autonomous mobile platforms require more human identity anonymized data to power their data-driven algorithms. The human identity anonymization should be robust so that less manual intervention is needed, which remains a challenge for current face detection and anonymization systems. In this paper, we propose to use the skeleton generated from the state-of-the-art human pose estimation model to help localize human heads. We develop criteria to evaluate the performance and compare it with the face detection approach. We demonstrate that the proposed algorithm can reduce missed face…
Controllable Localized Face Anonymization Via Diffusion Inpainting
Center for Machine Vision and Signal Analysis (CMVS)（机器视觉与信号分析中心）；University of Oulu（奥卢大学）；ELLIS Institute Finland（芬兰ELLIS研究所）；The Machine Learning Group（机器学习小组）；UiT The Arctic University of Norway（挪威北冰洋大学）
Beyond Inference Intervention: Identity-Decoupled Diffusion for Face Anonymization
School of Computer Science and Engineering, South China University of Technology（华南理工大学计算机科学与工程学院）；School of Future Technology, South China University of Technology（华南理工大学未来技术学院）；School of Computing and Information Systems, Singapore Management University（新加坡管理学院计算机与信息学院）
Cameras are prevalent in our daily lives, and enable many useful systems built upon computer vision technologies such as smart cameras and home robots for service applications. However, there is also an increasing societal concern as the captured images/videos may contain privacy-sensitive information (e.g., face identity). We propose a novel face identity transformer which enables automated photo-realistic password-based anonymization as well as deanonymization of human faces appearing in visual data. Our face identity transformer is trained to (1) remove face identity information after anony…
Facial Identity Anonymization via Intrinsic and Extrinsic Attention Distraction
School of Computer Science, Hangzhou Dianzi University（杭州电子科技大学计算机学院）；Department of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算机科学与技术学院）
Gender classification aims at recognizing a person's gender. Despite the high accuracy achieved by state-of-the-art methods for this task, there is still room for improvement in generalized and unrestricted datasets. In this paper, we advocate a new strategy inspired by the behavior of humans in gender recognition. Instead of dealing with the face image as a sole feature, we rely on the combination of isolated facial features and a holistic feature which we call the foggy face. Then, we use these features to train deep convolutional neural networks followed by an AdaBoost-based score fusion to…
Norwegian University of Science and Technology（挪威科技大学）
We propose a novel architecture which is able to automatically anonymize faces in images while retaining the original data distribution. We ensure total anonymization of all faces in an image by generating images exclusively on privacy-safe information. Our model is based on a conditional generative adversarial network, generating images considering the original pose and image background. The conditional information enables us to generate highly realistic faces with a seamless transition between the generated face and the existing background. Furthermore, we introduce a diverse dataset of huma…
Queen Mary University of London（伦敦玛丽女王大学）；University of Trento（特伦托大学）
This work addresses the problem of anonymizing the identity of faces in a dataset of images, such that the privacy of those depicted is not violated, while at the same time the dataset is useful for downstream task such as for training machine learning models. To the best of our knowledge, we are the first to explicitly address this issue and deal with two major drawbacks of the existing state-of-the-art approaches, namely that they (i) require the costly training of additional, purpose-trained neural networks, and/or (ii) fail to retain the facial attributes of the original images in the anon…
Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS（中国科学院计算技术研究所智能信息处理重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）；Michigan State University（密歇根州立大学）
Face attribute estimation has many potential applications in video surveillance, face retrieval, and social media. While a number of methods have been proposed for face attribute estimation, most of them did not explicitly consider the attribute correlation and heterogeneity (e.g., ordinal vs. nominal and holistic vs. local) during feature representation learning. In this paper, we present a Deep Multi-Task Learning (DMTL) approach to jointly estimate multiple heterogeneous attributes from a single face image. In DMTL, we tackle attribute correlation and heterogeneity with convolutional neural…
Technical University Munich（慕尼黑工业大学）；University of Venice（威尼斯大学）
The unprecedented increase in the usage of computer vision technology in society goes hand in hand with an increased concern in data privacy. In many real-world scenarios like people tracking or action recognition, it is important to be able to process the data while taking careful consideration in protecting people's identity. We propose and develop CIAGAN, a model for image and video anonymization based on conditional generative adversarial networks. Our model is able to remove the identifying characteristics of faces and bodies while producing high-quality images and videos that can be used…
Assessing the Use of Face Swapping Methods as Face Anonymizers in Videos
Department of Computer Engineering, Istanbul Technical University, Istanbul, Turkiye（伊斯坦布尔技术大学计算机工程系）；Division of Engineering, NYU Abu Dhabi, Abu Dhabi, UAE（纽约大学阿布扎比分校工程系）
Recently, productization of face recognition and identification algorithms have become the most controversial topic about ethical AI. As new policies around digital identities are formed, we introduce three face access models in a hypothetical social network, where the user has the power to only appear in photos they approve. Our approach eclipses current tagging systems and replaces unapproved faces with quantitatively dissimilar deepfakes. In addition, we propose new metrics specific for this task, where the deepfake is generated at random with a guaranteed dissimilarity. We explain access m…
The Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University（上海交通大学图像通信与网络工程研究所）；School of Computer Science, University of Technology Sydney（悉尼科技大学计算机科学学院）；Data61
Because of the explosive growth of face photos as well as their widespread dissemination and easy accessibility in social media, the security and privacy of personal identity information becomes an unprecedented challenge. Meanwhile, the convenience brought by advanced identity-agnostic computer vision technologies is attractive. Therefore, it is important to use face images while taking careful consideration in protecting people's identities. Given a face image, face de-identification, also known as face anonymization, refers to generating another image with similar appearance and the same ba…
University of La Rochelle（拉罗谢尔大学）；RIKEN（理化学研究所）；University of Toulouse（图卢兹大学）
This paper proposes a holistic multi-task Convolutional Neural Networks (CNNs) with the dynamic weights of the tasks,namely FaceLiveNet+, for face authentication. FaceLiveNet+ can employ face verification and facial expression recognition as a solution of liveness control simultaneously. Comparing to the single-task learning, the proposed multi-task learning can better capture the feature representation for all of the tasks. The experimental results show the superiority of the multi-task learning to the single-task learning for both the face verification task and facial expression recognition …
Anonymization of medical images is necessary for protecting the identity of the test subjects, and is therefore an essential step in data sharing. However, recent developments in deep learning may raise the bar on the amount of distortion that needs to be applied to guarantee anonymity. To test such possibilities, we have applied the novel CycleGAN unsupervised image-to-image translation framework on sagittal slices of T1 MR images, in order to reconstruct facial features from anonymized data. We applied the CycleGAN framework on both face-blurred and face-removed images. Our results show that…
Audi AG（奥迪股份公司）；Technische Hochschule Ingolstadt（因戈尔施塔特应用技术大学）
This work tackles the problem of temporally coherent face anonymization in natural video streams.We propose JaGAN, a two-stage system starting with detecting and masking out faces with black image patches in all individual frames of the video. The second stage leverages a privacy-preserving Video Generative Adversarial Network designed to inpaint the missing image patches with artificially generated faces. Our initial experiments reveal that image based generative models are not capable of inpainting patches showing temporal coherent appearance across neighboring video frames. To address this …
Ulsan National Institute of Science and Technology（蔚山科学技术院）；Korea Institute of Science and Technology（韩国科学技术研究院）
The robot market has been growing significantly and is expected to become 1.5 times larger in 2024 than what it was in 2019. Robots have attracted attention of security companies thanks to their mobility. These days, for security robots, unmanned aerial vehicles (UAVs) have quickly emerged by highlighting their advantage: they can even go to any hazardous place that humans cannot access. For UAVs, Drone has been a representative model and has several merits to consist of various sensors such as high-resolution cameras. Therefore, Drone is the most suitable as a mobile surveillance robot. These…
Faces are a class of visual stimuli with unique significance, for a variety of reasons. They are ubiquitous throughout the course of a person's life, and face recognition is crucial for daily social interaction. Faces are also unlike any other stimulus class in terms of certain physical stimulus characteristics. Furthermore, faces have been empirically found to elicit certain characteristic behavioral phenomena, which are widely held to be evidence of "holistic" processing of faces. However, little is known about the neural mechanisms underlying such holistic face processing. In other words, f…
Face anonymization with generative models have become increasingly prevalent since they sanitize private information by generating virtual face images, ensuring both privacy and image utility. Such virtual face images are usually not identifiable after the removal or protection of the original identity. In this paper, we formalize and tackle the problem of generating identifiable virtual face images. Our virtual face images are visually different from the original ones for privacy protection. In addition, they are bound with new virtual identities, which can be directly used for face recogniti…
Privacy of machine learning models is one of the remaining challenges that hinder the broad adoption of Artificial Intelligent (AI). This paper considers this problem in the context of image datasets containing faces. Anonymization of such datasets is becoming increasingly important due to their central role in the training of autonomous cars, for example, and the vast amount of data generated by surveillance systems. While most prior work de-identifies facial images by modifying identity features in pixel space, we instead project the image onto the latent space of a Generative Adversarial Ne…
Face verification and recognition problems have seen rapid progress in recent years, however recognition from small size images remains a challenging task that is inherently intertwined with the task of face super-resolution. Tackling this problem using multiple frames is an attractive idea, yet requires solving the alignment problem that is also challenging for low-resolution faces. Here we present a holistic system for multi-frame recognition, alignment, and superresolution of faces. Our neural network architecture restores the central frame of each input sequence additionally taking into ac…
