This paper revisits Deep Mutual Learning (DML), a simple yet effective computing paradigm. We propose using Rényi divergence instead of the KL divergence, which is more flexible and tunable, to improve vanilla DML. This modification is able to consistently improve performance over vanilla DML with limited additional complexity. The convergence properties of the proposed paradigm are analyzed theoretically, and Stochastic Gradient Descent with a constant learning rate is shown to converge with $\mathcal{O}(1)$-bias in the worst case scenario for nonconvex optimization tasks. That is, learning will reach nearby local optima but continue searching within a bounded scope, which may help mitigate overfitting. Finally, our extensive empirical results demonstrate the advantage of combining DML and Rényi divergence, leading to further improvement in model generalization.
Tighter Information-Theoretic Generalization Bounds via a Novel Class of Change of Measure Inequalities
Department of Electrical and Electronic Engineering, Imperial College London(帝国理工学院伦敦分校电子与电气工程系);Department of Information Engineering, The Chinese University of Hong Kong(香港中文大学信息工程系)
Reliable Estimation of KL Divergence using a Discriminator in Reproducing Kernel Hilbert Space
Northeastern University(东北大学);Department of Electrical and Computer Engineering(电气与计算机工程系);Boston, MA, USA(美国马萨诸塞州波士顿(此处按要求不提取,实际应为无,原文本中此条是地址,非机构,正确应只取Northeastern University和Department of Electrical and Computer Engineering,但Department of Electrical and Computer Engineering是系,属于大学内部分类,按要求粒度到大学/学院/公司/研究院级别即可,所以只保留Northeastern University)}]}修正:根据提取规则,机构粒度到大学/学院/公司/研究院级别即可,不要细拆到系,所以正确应为{)
Transformers Provably Learn Directed Acyclic Graphs via Kernel-Guided Mutual Information
Department of Mathematics, National University of Singapore, Singapore(新加坡国立大学数学系);Department of Statistics and Data Science, University of Pennsylvania, USA(宾夕法尼亚大学统计与数据科学系);Department of Electrical and Computer Engineering, The Ohio State University, USA(俄亥俄州立大学电子与计算机工程系)
Generalised Mutual Information for Discriminative Clustering
Université Côte d’Azur(蔚蓝海岸大学);Inria(法国国家信息与自动化研究所);CNRS(法国国家科学研究中心);CHU de Québec Research Center(魁北克大学医院研究中心);Laval University(拉瓦尔大学);Jellysmack;AI Labs(人工智能实验室)
$f$-MICL: Understanding and Generalizing InfoNCE-based Contrastive Learning
School of Computer Science(计算机科学学院);University of Waterloo(滑铁卢大学);Vector Institute(矢量研究院);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);National Research Council Canada(加拿大国家研究委员会);University of Ottawa(渥太华大学)
Generalised Mutual Information: a Framework for Discriminative Clustering
Université Côte d’Azur(蔚蓝海岸大学);INRIA(法国国家信息与自动化研究所);CNRS(法国国家科学研究中心);Université Laval(拉瓦尔大学);CHU de Québec Research Centre(魁北克大学医院研究中心);Jellysmack(Jellysmack公司)