arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.20545stat.MLcs.LG
收藏

迁移学习的样本复杂性:一种最优传输方法

Sample Complexity of Transfer Learning: An Optimal Transport Approach

Tsinghua-Berkeley Shenzhen Institute(清华大学-伯克利深圳研究院)

Haoyang Cao, Xin Guo, Wenpin Tang, Guan Wang

AI总结:

本文通过最优传输视角分析迁移学习的样本效率,发现当数据维度d大于3时,迁移学习的样本复杂性为O(m^{-(α+1)/d}),优于直接学习的O(m^{-p/d}),其中α表示数据分布的光滑度,p表示最优目标模型的光滑度。

摘要:

迁移学习是许多复杂结构的机器学习/AI模型,如大语言模型和生成式AI中的关键技术。迁移学习的本质是利用已解决的源任务知识来解决新目标任务,尤其是在后者训练数据样本量m较低时。本文严格分析了迁移学习在样本效率方面的潜在优势。具体而言,从最优传输视角出发,我们发现当数据维度d大于3时,迁移学习的样本复杂性为O(m^{-(α+1)/d}),其中α表示数据分布的光滑度,而直接学习的样本复杂性为O(m^{-p/d}),其中p表示最优目标模型的光滑度。我们的发现从理论上支持了当目标任务在一系列不太光滑的模型(即高度复杂的网络,可能使用非光滑激活函数)中优化时,迁移学习具有更好的样本效率。以图像分类为例,我们通过数值实验展示了迁移学习的样本效率,即在数据渴求的 regime 中,迁移学习可以显著提升模型性能。

英文摘要:

Transfer learning is an essential technique for many machine learning/AI models of complex structures such as large language models and generative AI. The essence of transfer learning is to leverage knowledge from resolved source tasks for a new target task, especially when the sample size $m$ of the training data for the latter is low. In this work, we rigorously analyze the potential benefit of transfer learning in terms of sample efficiency. Specifically, taking an optimal transport viewpoint of transfer learning, we find that when the data dimension $d$ is higher than $3$, the sample complexity for transfer learning is $O(m^{-(α+1)/d})$, with $α$ indicating the smoothness of the data distribution, as opposed to the $O(m^{-p/d})$ sample complexity for direct learning with $p$ indicating the smoothness of the optimal target model. Our finding theoretically supports a better sample efficiency for transfer learning, when the target task is optimizing over a family of not-so-smooth models (i.e., highly complex networks with the possible use of non-smooth activation functions). Using image classification as an example, we numerically demonstrate the sample efficiency for transfer learning, that is, in the data hungry regime, the model performance can be significantly improved by transfer learning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑