arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2311.14756cs.LGcs.AI
收藏

任务分布鲁棒数据免费元学习

Task-Distributionally Robust Data-Free Meta-Learning

College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算机与数据科学学院);School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区网络空间安全学院);Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Department of Computer Science and AI Institute, University of Central Florida(中佛罗里达大学计算机科学系与人工智能研究所);School of Data Science, the Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen) and Secure Computing Lab of Big Data, Shenzhen Research Institute of Big Data (SBRID)(香港中文大学(深圳)数据科学学院与深圳市大数据研究院大数据安全计算实验室)

Zixuan Hu, Yongxian Wei, Li Shen, Zhenyi Wang, Baoyuan Wu, Chun Yuan, Dacheng Tao

AI总结:

本文研究数据免费元学习的鲁棒性,发现任务分布偏移和任务分布污染两种漏洞,并提出可信DFML框架以缓解这些问题。

摘要:

数据免费元学习(DFML)旨在通过从多个预训练模型中元学习来高效学习未见过的少样本任务,而无需访问其原始训练数据。尽管现有DFML方法通常通过生成合成数据进行元学习,但对其鲁棒性的全面分析,尤其是其失败模式和对潜在攻击的脆弱性,仍然明显缺失。本文通过系统研究DFML的鲁棒性,识别出两个关键但此前被忽视的漏洞:任务分布偏移(TDS)和任务分布污染(TDC)。TDS指的是任务分布的连续偏移,导致先前学习的元知识灾难性遗忘。TDC暴露了DFML的安全漏洞,揭示了当预训练模型池包含不可信模型时,其易受攻击的特性。为缓解这些漏洞,我们提出一个可信DFML框架,包含三个组件:合成任务重建、元学习与任务记忆插值以及自动模型选择。具体而言,利用模型逆向技术,从多个预训练模型中重建合成任务以进行元学习。为防止遗忘,我们引入一种策略以回放插值的历史任务以高效回忆先前的元知识。此外,我们的框架无缝整合了自动模型选择机制,以在元学习过程中自动过滤掉不可信的模型。代码可在https://github.com/Egg-Hu/Trustworthy-DFML获取。

英文摘要:

Data-Free Meta-Learning (DFML) aims to enable efficient learning of unseen few-shot tasks, by meta-learning from multiple pre-trained models without accessing their original training data. While existing DFML methods typically generate synthetic data from these models to perform meta-learning, a comprehensive analysis of DFML's robustness-particularly its failure modes and vulnerability to potential attacks-remains notably absent. Such an analysis is crucial as algorithms often operate in complex and uncertain real-world environments. This paper fills this significant gap by systematically investigating the robustness of DFML, identifying two critical but previously overlooked vulnerabilities: Task-Distribution Shift (TDS) and Task-Distribution Corruption (TDC). TDS refers to the sequential shifts in the evolving task distribution, leading to the catastrophic forgetting of previously learned meta-knowledge. TDC exposes a security flaw of DFML, revealing its susceptibility to attacks when the pre-trained model pool includes untrustworthy models that deceptively claim to be beneficial but are actually harmful. To mitigate these vulnerabilities, we propose a trustworthy DFML framework comprising three components: synthetic task reconstruction, meta-learning with task memory interpolation, and automatic model selection. Specifically, utilizing model inversion techniques, we reconstruct synthetic tasks from multiple pre-trained models to perform meta-learning. To prevent forgetting, we introduce a strategy to replay interpolated historical tasks to efficiently recall previous meta-knowledge. Furthermore, our framework seamlessly incorporates an automatic model selection mechanism to automatically filter out untrustworthy models during the meta-learning process. Code is available at https://github.com/Egg-Hu/Trustworthy-DFML.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑