We propose an efficient neural network for RAW image denoising. Although neural network-based denoising has been extensively studied for image restoration, little attention has been given to efficient denoising for compute limited and power sensitive devices, such as smartphones and smartwatches. In this paper, we present a novel architecture and a suite of training techniques for high quality denoising in mobile devices. Our work is distinguished by three main contributions. (1) Feature-Align layer that modulates the activations of an encoder-decoder architecture with the input noisy images. The auto modulation layer enforces attention to spatially varying noise that tend to be "washed away" by successive application of convolutions and non-linearity. (2) A novel Feature Matching Loss that allows knowledge distillation from large denoising networks in the form of a perceptual content loss. (3) Empirical analysis of our efficient model trained to specialize on different noise subranges. This opens additional avenue for model size reduction by sacrificing memory for compute. Extensive experimental validation shows that our efficient model produces high quality denoising results that compete with state-of-the-art large networks, while using significantly fewer parameters and MACs. On the Darmstadt Noise Dataset benchmark, we achieve a PSNR of 48.28dB, while using 263 times fewer MACs, and 17.6 times fewer parameters than the state-of-the-art network, which achieves 49.12dB.
Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed
诸如噪声条件得分网络和去噪扩散概率模型的迭代生成模型通过逐渐对初始噪声向量去噪来产生高质量样本。然而，它们的去噪过程有很多步骤，使它们比其他生成模型（如GANs和VAE）慢2-3个数量级。在本文中，我们建立了知识蒸馏和图像生成之间的一种新的连接，该技术将多步去噪过程蒸馏为单步的，从而获得与其他单步生成模型相似的采样速度。我们的去噪学生在CIFAR-10和CelebA数据集上生成与GANs相当的高质量样本，而无需对抗训练。我们证明了我们的方法通过在256 x 256 LSUN上的实验可以扩展到更高的分辨率。代码和检查点可在https://github.com/tcl9876/Denoising_Student上获得
Denoising Mutual Knowledge Distillation in Bi-Directional Multiple Instance Learning
College of Computer Science, Sichuan University（四川大学计算机学院）；Sichuan University-Pittsburgh Institute（四川大学匹兹堡研究院）；Department of Pathology West China Hospital, Sichuan University（四川大学华西医院病理科部门）；Institute of Clinical Pathology, West China Hospital, Sichuan University（四川大学华西医院临床病理研究所）
Omnimodal Dataset Distillation via High-order Proxy Alignment
University of Science and Technology of China（中国科学技术大学）；Xidian University（西安电子科技大学）；National University of Singapore（新加坡国立大学）；The University of Sydney（悉尼大学）
Decoupling Dark Knowledge via Block-wise Logit Distillation for Feature-level Alignment
College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息与电子工程学院）；ZJU-UIUC Institute, Zhejiang University（浙江大学-伊利诺伊大学厄巴纳-香槟分校联合学院）
AgentPose: Progressive Distribution Alignment via Feature Agent for Human Pose Distillation
Nanjing University of Posts and Telecommunications（南京邮电大学）；Surrey Institute for People-Centred Artificial Intelligence, University of Surrey（萨里大学以人为本人工智能研究所）
Not All Timesteps Matter Equally: Selective Alignment Knowledge Distillation for Spiking Neural Networks
Faculty of Information Technolody, Monash University, Australia（墨尔本大学信息科技学院，澳大利亚）；School of Software, Northeastern University, China（东北大学软件学院，中国）；Department of Medicine, National University of Singapore, Singapore（新加坡国立大学医学部，新加坡）
Holistic White-light Polyp Classification via Alignment-free Dense Distillation of Auxiliary Optical Chromoendoscopy
Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology（光电国立实验室、华中科技大学）；Changzhou United Imaging Surgical Co., Ltd（常州联合影像手术有限公司）；Tongji Medical College of Huazhong University of Science and Technology（华中科技大学同济医学院）
Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models
Pingtan Research Institute of Xiamen University（厦门大学滨海研究院）；Imperial College London（伦敦帝国理工学院）；University of Sussex（苏塞克斯大学）；Purdue University（普渡大学）；University of Liverpool（利物浦大学）；JTB Technology Corp.（JTB技术公司）；Emory University（埃默里大学）；The University of Texas at Dallas（德克萨斯大学达拉斯分校）；Kyoto University（京都大学）；AppCubic；Georgia Institute of Technology（佐治亚理工学院）；AI Agent Lab（AI代理实验室）
Multiple Degradation and Reconstruction Network for Single Image Denoising via Knowledge Distillation
The Chinese University of Hong Kong（香港中文大学）；East China Normal University（华东师范大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）
随着深度学习的发展，单幅图像去噪（SID）取得了重大突破。然而，所提出的方法往往伴随着大量的参数，这极大地限制了它们的应用场景。与以往盲目增加网络深度的方法不同，本文研究了含噪图像的退化机制，提出了一种轻量级的多重退化与重构网络（Multiple Degradation and Reconstruction Network，MDRN）来逐步去除噪声。同时，我们提出了两种新的异构知识蒸馏策略（HMDS），使MDRN学习更丰富，更准确的特征，从异构模型，这使得它有可能在极端条件下重建更高质量的去噪图像。大量的实验表明，我们的MDRN实现了良好的性能对其他SID模型与较少的参数。同时，大量的烧蚀研究表明，引入HMDS可以改善微小模型或高噪声水平下模型的性能，这在相关应用中是非常有用的。
