Rethinking Vision Transformer and Masked Autoencoder in Multimodal Face Anti-Spoofing
Great Bay University（大湾区大学）；Nanyang Technological University（南洋理工大学）；University of Oulu（奥卢大学）；Lappeenranta-Lahti University of Technology（拉彭兰塔-拉赫蒂理工大学）；South China University of Technology（华南理工大学）
Zitong Yu, Rizhao Cai, Yawen Cui, Xin Liu, Yongjian Hu, Alex Kot
Recently, vision transformer (ViT) based multimodal learning methods have been proposed to improve the robustness of face anti-spoofing (FAS) systems. However, there are still no works to explore the fundamental natures (\textit{e.g.}, modality-aware inputs, suitable multimodal pre-training, and efficient finetuning) in vanilla ViT for multimodal FAS. In this paper, we investigate three key factors (i.e., inputs, pre-training, and finetuning) in ViT for multimodal FAS with RGB, Infrared (IR), and Depth. First, in terms of the ViT inputs, we find that leveraging local feature descriptors benefits the ViT on IR modality but not RGB or Depth modalities. Second, in observation of the inefficiency on direct finetuning the whole or partial ViT, we design an adaptive multimodal adapter (AMA), which can efficiently aggregate local multimodal features while freezing majority of ViT parameters. Finally, in consideration of the task (FAS vs. generic object classification) and modality (multimodal vs. unimodal) gaps, ImageNet pre-trained models might be sub-optimal for the multimodal FAS task. To bridge these gaps, we propose the modality-asymmetric masked autoencoder (M$^{2}$A$^{2}$E) for multimodal FAS self-supervised pre-training without costly annotated labels. Compared with the previous modality-symmetric autoencoder, the proposed M$^{2}$A$^{2}$E is able to learn more intrinsic task-aware representation and compatible with modality-agnostic (e.g., unimodal, bimodal, and trimodal) downstream settings. Extensive experiments with both unimodal (RGB, Depth, IR) and multimodal (RGB+Depth, RGB+IR, Depth+IR, RGB+Depth+IR) settings conducted on multimodal FAS benchmarks demonstrate the superior performance of the proposed methods. We hope these findings and solutions can facilitate the future research for ViT-based multimodal FAS.
Benefiting from masked visual modeling, self-supervised video representation learning has achieved remarkable progress. However, existing methods focus on learning representations from scratch through reconstructing low-level features like raw pixel RGB values. In this paper, we propose masked video distillation (MVD), a simple yet effective two-stage masked feature modeling framework for video representation learning: firstly we pretrain an image (or video) model by recovering low-level features of masked patches, then we use the resulting features as targets for masked feature modeling. For …
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Google Research（谷歌研究院）；Michigan State University（密歇根州立大学）
This work studies the generalization issue of face anti-spoofing (FAS) models on domain gaps, such as image resolution, blurriness and sensor variations. Most prior works regard domain-specific signals as a negative impact, and apply metric learning or adversarial losses to remove them from feature representation. Though learning a domain-invariant feature space is viable for the training data, we show that the feature shift still exists in an unseen test domain, which backfires on the generalizability of the classifier. In this work, instead of constructing a domain-invariant feature space, w…
Rethinking Data Distillation: Do Not Overlook Calibration
Texas A&M University（德州农工大学）；Zhejiang University（浙江大学）；Certik；University of Maryland, College Park（马里兰大学帕克分校）；Purdue University（普渡大学）；North Carolina State University（北卡罗来纳州立大学）
A Unified Framework for Iris Anti-Spoofing: Introducing Iris Anti-Spoofing Cross-Domain-Testing Protocol and Masked-MoE Method
China Telecom Research Institute（中国电信研究院）；Southern University of Science and Technology (SUSTech)（南方科技大学）；Shenzhen Institutes of Advanced Technology (SIAT)（深圳先进技术研究院）；Shenzhen University of Advanced Technology (SUAT)（深圳理工大学）；Macau University of Science and Technology（澳门科技大学）；Tianjin University of Science & Technology（天津科技大学）
Purify then Guide: Rethinking Domain Generalization for Multimodal Face Anti-Spoofing
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；School of Computing and Information Technology, Great Bay University（大湾区大学计算与信息科技学院）；Dongguan Key Laboratory for Intelligence and Information Technology（东莞智能与信息处理重点实验室）；Guangdong Provincial Key Laboratory of Intelligent Information Processing, Shenzhen University（广东省智能信息处理重点实验室）；Tencent Youtu Lab（腾讯优图实验室）；School of Psychology, Shanghai Jiao Tong University（上海交通大学心理学院）；Faculty of Applied Sciences, Macao Polytechnic University（澳门理工学院应用科学学院）；The Chinese University of Hong Kong（香港中文大学）
Shanghai AI Laboratory（上海人工智能实验室）；Simon Fraser University（西蒙弗雷泽大学）；The Chinese University of Hong Kong（香港中文大学）
Masked Image Modeling (MIM) has achieved promising progress with the advent of Masked Autoencoders (MAE) and BEiT. However, subsequent works have complicated the framework with new auxiliary tasks or extra pre-trained models, inevitably increasing computational overhead. This paper undertakes a fundamental analysis of MIM from the perspective of pixel reconstruction, which examines the input image patches and reconstruction target, and highlights two critical but previously overlooked bottlenecks. Based on this analysis, we propose a remarkably simple and effective method, {\ourmethod}, that e…
FSFM: A Generalizable Face Security Foundation Model via Self-Supervised Facial Representation Learning
State Key Laboratory of Blockchain and Data Security（区块链与数据安全国家重点实验室）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江区）区块链与数据安全研究院）
JD Explore Academy（京东探索研究院）；The University of Sydney（悉尼大学）
Self-supervised pre-training vision transformer (ViT) via masked image modeling (MIM) has been proven very effective. However, customized algorithms should be carefully designed for the hierarchical ViTs, e.g., GreenMIM, instead of using the vanilla and simple MAE for the plain ViT. More importantly, since these hierarchical ViTs cannot reuse the off-the-shelf pre-trained weights of the plain ViTs, the requirement of pre-training them leads to a massive amount of computational cost, thereby incurring both algorithmic and computational complexity. In this paper, we address this problem by propo…
Rethinking Point Clouds as Sequences: A Causal Next-Token Predictive Learning Framework
Nanjing University of Science and Technology（南京理工大学）；Shanghai Jiao Tong University（上海交通大学）；Hangzhou City University（杭州城市学院）；East China Normal University（华东师范大学）
