Grouped Knowledge Distillation for Deep Face Recognition
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences（中国科学院香港创新研究院人工智能与机器人中心）
Weisong Zhao, Xiangyu Zhu, Kaiwen Guo, Xiao-Yu Zhang, Zhen Lei
备注：
9 pages, 2 figures, 7 tables, accepted by AAAI 2023
Compared with the feature-based distillation methods, logits distillation can liberalize the requirements of consistent feature dimension between teacher and student networks, while the performance is deemed inferior in face recognition. One major challenge is that the light-weight student network has difficulty fitting the target logits due to its low model capacity, which is attributed to the significant number of identities in face recognition. Therefore, we seek to probe the target logits to extract the primary knowledge related to face identity, and discard the others, to make the distillation more achievable for the student network. Specifically, there is a tail group with near-zero values in the prediction, containing minor knowledge for distillation. To provide a clear perspective of its impact, we first partition the logits into two groups, i.e., Primary Group and Secondary Group, according to the cumulative probability of the softened prediction. Then, we reorganize the Knowledge Distillation (KD) loss of grouped logits into three parts, i.e., Primary-KD, Secondary-KD, and Binary-KD. Primary-KD refers to distilling the primary knowledge from the teacher, Secondary-KD aims to refine minor knowledge but increases the difficulty of distillation, and Binary-KD ensures the consistency of knowledge distribution between teacher and student. We experimentally found that (1) Primary-KD and Binary-KD are indispensable for KD, and (2) Secondary-KD is the culprit restricting KD at the bottleneck. Therefore, we propose a Grouped Knowledge Distillation (GKD) that retains the Primary-KD and Binary-KD but omits Secondary-KD in the ultimate KD loss calculation. Extensive experimental results on popular face recognition benchmarks demonstrate the superiority of proposed GKD over state-of-the-art methods.
Dual Teacher Knowledge Distillation with Domain Alignment for Face Anti-spoofing
Sun Yat-sen University（中山大学）；Tencent YouTu Lab（腾讯优图实验室）；Nanjing University（南京大学）；Harbin Institute of Technology（哈尔滨工业大学）；Southern University of Science and Technology（南方科技大学）
Synthetic Data Generation for Emotional Depth Faces: Optimizing Conditional DCGANs via Genetic Algorithms in the Latent Space and Stabilizing Training with Knowledge Distillation
Cyrus Intelligence Research Ltd（居鲁士智能研究有限公司）；Pars AI
Cross-Resolution Face Recognition via Prior-Aided Face Hallucination and Residual Knowledge Distillation
Xi’an Jiaotong University（西安交通大学）；National University of Singapore（新加坡国立大学）；National University of Defense Technology（中国人民解放军国防科技大学）；Pensees Pte Ltd（Pensees私人有限公司）；University of Electronic Science and Technology of China（电子科技大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
How Knowledge Distillation Mitigates the Synthetic Gap in Fair Face Recognition
INESC TEC；Faculty of Engineering of the University of Porto（波尔图大学工程学院）；Faculty of Electrical Engineering and Computer Science, University of Maribor（马里博尔大学电气工程与计算机科学学院）
Not Every Cue Is Evidence: Cue Supportability for Forensic Distillation in Face Forgery Detection
School of Software Technology, Zhejiang University（浙江大学软件学院）；Ant Group（蚂蚁集团）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；College of Computer Science and Technology, Zhejiang University of Technology（浙江工业大学计算机科学与技术学院）
Prior Knowledge Distillation Network for Face Super-Resolution
School of Computer and Information, Hefei University of Technology（合肥工业大学计算机与信息学院）；Institute of Advanced Technology, University of Science and Technology of China（中国科学技术大学先进技术研究院）
One-Class Knowledge Distillation for Face Presentation Attack Detection
Nanyang Technological University（南洋理工大学）；City University of Hong Kong（香港城市大学）；South China University of Technology（华南理工大学）；China-Singapore International Joint Research Institute（中新国际联合研究院）
Flow Augmentation and Knowledge Distillation for Lightweight Face Presentation Attack Detection
SDAIA-KFUPM Joint Research Center for Artificial Intelligence（SDAIA-KFUPM联合人工智能研究中心）；King Fahd University of Petroleum & Minerals（国王法赫德石油大学）；Interdisciplinary Research Center for Intelligent Secure Systems (IRC-ISS)（智能安全系统跨学科研究中心）；College of Information Science & Electronic Engineering（信息科学与电子工程学院）；Department of Computer Engineering, College of Computing and Mathematics（计算机工程系，计算与数学学院）
Face to Cartoon Incremental Super-Resolution using Knowledge Distillation
Indian Institute of Information Technology, Allahabad（印度阿拉哈巴德信息技术学院）；Sorbonne Centre for Artificial Intelligence, Sorbonne University Abu Dhabi（索邦大学阿布扎比分校索邦人工智能中心）
Distill-2MD-MTL: Data Distillation based on Multi-Dataset Multi-Domain Multi-Task Frame Work to Solve Face Related Tasksks, Multi Task Learning, Semi-Supervised Learning
Bag of Instances Aggregation Boosts Self-supervised Distillation
Shanghai Jiao Tong University（上海交通大学）；Huawei Inc.（华为公司）；Huazhong University of Science & Technology（华中科技大学）
自监督学习的最新进展经历了显着的进步，特别是对于基于对比学习的方法，它将每个图像及其增强视为一个单独的类，并试图将它们与所有其他图像区分开来。然而，由于大量的样本，这种借口任务本质上遭受缓慢的收敛和难以优化。这对于小规模模型尤其如此，我们发现与监督模型相比，性能急剧下降。在本文中，我们提出了一个简单而有效的无监督学习的蒸馏策略。其亮点在于，相似样本之间的关系很重要，可以无缝地转移到学生身上，以提高成绩。我们的方法，被称为BINGO，是Bag of InstaNces aggregatiOn的缩写，旨在将教师学到的关系转移到学生身上。这里的实例包表示教师构建的一组相似样本，并分组在一个包中，蒸馏的目标是聚合学生关于包中实例的紧凑表示。值得注意的是，BINGO在小规模模型上实现了新的最先进性能，即，在ImageNet上分别使用ResNet-18和ResNet-34作为主干进行线性评估，获得了65.5%和68.9%的top-1准确率，显著超过了基线（52.5%和57.4%的top-1准确率）。该代码可在https://github.com/haohang96/bingo上获得。
Learning Visual Speech Representations via Cross-Modal Distillation and Joint Face-Lip Modeling
School of Artificial Intelligence and Computer Science, Shaanxi Normal University（陕西师范大学人工智能与计算机科学学院）；iFLYTEK Research, iFLYTEK Co., Ltd.（科大讯飞股份有限公司）；Key Laboratory of Intelligent Computing and Service Technology for Folk SongMinistry of Culture and Tourism（文化和旅游部民歌智能计算与服务技术重点实验室）
Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS)（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Horizon Robotics（地平线机器人）；Peng Cheng Laboratory（鹏城实验室）
CAKE: Real-time Action Detection via Motion Distillation and Background-aware Contrastive Learning
School of Electrical and Electronics Engineering, Hanoi University of Science and Technology, Vietnam（河内科技大学电气与电子工程学院，越南）；Signal Analysis and Interpretation Lab, University of Southern California, United States（南加州大学信号分析与解释实验室，美国）
