arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2108.11018cs.LGcs.CV
收藏

一个缩放定律的合成到真实的传输:多少是你的预培训有效?

A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your Pre-training Effective?

Preferred Networks, Inc.(优选网络公司);The Institute of Statistical Mathematics(统计数学研究所);The University of Tokyo(东京大学);AIP-RIKEN(理化学研究所先进智能项目)

Hiroaki Mikami, Kenji Fukumizu, Shogo Murai, Shuji Suzuki, Yuta Kikuchi, Taiji Suzuki, Shin-ichi Maeda, Kohei Hayashi

摘要:

合成到真实迁移学习是一种框架,其中合成生成的数据集用于预训练模型,以提高其在真实视觉任务上的性能。使用合成图像的最大优势是地面实况标签自动可用,从而可以在不增加人力成本的情况下无限扩展数据大小。然而,合成数据可能具有巨大的域间隙,在这种情况下,增加数据大小不会提高性能。我们怎么知道?在这项研究中,我们推导出一个简单的缩放定律,预测的性能从量的预训练数据。通过估计该规律的参数,可以判断是否应该增加数据量或改变图像合成的设置。此外,我们分析了迁移学习的理论,考虑到学习动力学,并确认派生的泛化界是与我们的实证研究结果是一致的。我们经验验证了我们的缩放定律的基准任务,模型大小和合成图像的复杂性的各种实验设置。

英文摘要:

Synthetic-to-real transfer learning is a framework in which a synthetically generated dataset is used to pre-train a model to improve its performance on real vision tasks. The most significant advantage of using synthetic images is that the ground-truth labels are automatically available, enabling unlimited expansion of the data size without human cost. However, synthetic data may have a huge domain gap, in which case increasing the data size does not improve the performance. How can we know that? In this study, we derive a simple scaling law that predicts the performance from the amount of pre-training data. By estimating the parameters of the law, we can judge whether we should increase the data or change the setting of image synthesis. Further, we analyze the theory of transfer learning by considering learning dynamics and confirm that the derived generalization bound is consistent with our empirical findings. We empirically validated our scaling law on various experimental settings of benchmark tasks, model sizes, and complexities of synthetic images.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑