Improving Heterogeneous Model Reuse by Density Estimation
Wuhan University(武汉大学);Hubei Luojia Laboratory(湖北珞珈实验室);Beijing Institute of Technology(北京理工大学);JD Explore Academy, JD.com, Inc.(京东集团京东探索研究院);Washington University in St. Louis(圣路易斯华盛顿大学);The University of Sydney(悉尼大学)
Anke Tang, Yong Luo, Han Hu, Fengxiang He, Kehua Su, Bo Du, Yixin Chen, Dacheng Tao
This paper studies multiparty learning, aiming to learn a model using the private data of different participants. Model reuse is a promising solution for multiparty learning, assuming that a local model has been trained for each party. Considering the potential sample selection bias among different parties, some heterogeneous model reuse approaches have been developed. However, although pre-trained local classifiers are utilized in these approaches, the characteristics of the local data are not well exploited. This motivates us to estimate the density of local data and design an auxiliary model together with the local classifiers for reuse. To address the scenarios where some local models are not well pre-trained, we further design a multiparty cross-entropy loss for calibration. Upon existing works, we address a challenging problem of heterogeneous model reuse from a decision theory perspective and take advantage of recent advances in density estimation. Experimental results on both synthetic and benchmark data demonstrate the superiority of the proposed method.
Protecting Deep Learning Model Copyrights with Adversarial Example-Free Reuse Detection
School of Mathematical Sciences, Peking University, China(北京大学数学科学学院);School of Computer Science, University of Bristol, UK(英国布里斯托大学计算机科学学院);School of Control Science and Engineering, Zhejiang University, China(浙江大学控制科学与工程学院)
提出 NFARD,利用神经元功能与正常样本检测 DNN 复用关系,支持白盒、黑盒及异构场景,并构建 Reuse Zoo,检测 F1 最高达 1.0。
Reduce, Reuse, Recycle: Improving Training Efficiency with Distillation
Texas State University(德克萨斯州立大学);MosaicML(MosaicML公司);Brown University(布朗大学)
提高深度网络训练效率的方法(即达到给定模型质量水平所需的资源)对深度学习实践者来说是直接有益的。蒸馏通常用于压缩模型或提高模型质量,但尚不清楚蒸馏是否真的提高了训练效率。蒸馏的质量改进是否可以转化为训练的加速,或者它们只是提高了最终模型的质量而没有节省资源?我们进行了一系列实验,以研究蒸馏是否以及如何使用ImageNet上训练的ResNet-50和C4上训练的BERT来加速训练,并使用通用企业硬件(8 x NVIDIA A100)进行GLUE评估。我们发现,在ImageNet上训练的ResNet-50中,蒸馏可以将训练速度提高1.96倍,在GLUE上评估时,在BERT上可以提高1.42倍。此外,BERT的蒸馏仅在训练的前20-50%执行时产生最佳结果。我们还观察到,在ResNet-50和BERT中,使用蒸馏的训练几乎总是比不使用蒸馏的训练更有效,即使使用质量最差的模型作为老师。最后,我们发现,通过在每个步骤上从教师模型池中随机抽样一个教师,可以从具有O(n)运行时成本的教师模型集合中获得提取的好处,这只有O(1)运行时成本。综上所述,这些结果表明,蒸馏可以大大提高图像分类和语言建模的训练效率,并且对蒸馏协议进行一些简单的优化可以进一步提高这些效率。
Bifrost: Steering Strategic Trajectories to Bridge Contextual Gaps for Self-Improving Agents
The University of Sydney(悉尼大学);Florida International University(佛罗里达国际大学);Hong Kong Baptist University(香港浸会大学);RIKEN Center for Advanced Intelligence Project(理化学研究所高级智能项目中心);The University of Tokyo(东京大学)
FedBR: Improving Federated Learning on Heterogeneous Data via Local Learning Bias Reduction
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));The Shenzhen Institute of Artificial Intelligence and Robotics for Society(深圳市人工智能与机器人研究院);Westlake University(西湖大学)
Adaptive Bi-Recommendation and Self-Improving Network for Heterogeneous Domain Adaptation-Assisted IoT Intrusion Detection
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);University of Macau(澳门大学);University of New Brunswick(新不伦瑞克大学)
Understanding and Improving Model Averaging in Federated Learning on Heterogeneous Data
Academy of Interdisciplinary Studies, The Hong Kong University of Science and Technology(香港科技大学跨学科研究学院);The Hong Kong University of Science and Technology(香港科技大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))