arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2301.02064cs.CVcs.AI
收藏

使用视觉Transformer的单轮自监督分布式学习

Single-round Self-supervised Distributed Learning using Vision Transformer

Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院);Yonsei University College of Medicine(延世大学医学院);Gangnam Severance Hospital(江南世福兰斯医院)

Sangjoon Park, Ik-Jae Lee, Jun Won Kim, Jong Chul Ye

AI总结:

本文提出一种基于视觉Transformer的自监督掩码采样蒸馏方法,在单轮通信内实现隐私增强的分布式学习,并在医学图像任务上验证了其优于现有分布式学习策略的性能及作为任务无关基础模型的潜力。

摘要:

尽管深度学习近年来在医学领域取得了成功,但数据稀缺问题因隐私和数据所有权的担忧而进一步加剧。包括联邦学习在内的分布式学习方法已被研究以解决这些问题。然而,它们受到繁重通信开销的需求以及隐私保护方面的弱点的阻碍。为了应对这些挑战,我们提出了一种针对视觉Transformer的自监督掩码采样蒸馏方法。该方法可以在无需持续通信的情况下实现,并可通过利用视觉Transformer特有的加密技术来增强隐私。我们在两个不同的任务上进行了广泛的实验,证明了我们方法的有效性。与现有的分布式学习策略以及仅微调的基线相比,我们取得了更优越的性能。此外,由于使用我们提出的方法创建的自监督模型能够实现对图像的一般语义理解,我们展示了其作为面向各种下游任务的任务无关自监督基础模型的潜力,从而扩展了其在医学领域的适用性。

英文摘要:

Despite the recent success of deep learning in the field of medicine, the issue of data scarcity is exacerbated by concerns about privacy and data ownership. Distributed learning approaches, including federated learning, have been investigated to address these issues. However, they are hindered by the need for cumbersome communication overheads and weaknesses in privacy protection. To tackle these challenges, we propose a self-supervised masked sampling distillation method for the vision transformer. This method can be implemented without continuous communication and can enhance privacy by utilizing a vision transformer-specific encryption technique. We conducted extensive experiments on two different tasks, which demonstrated the effectiveness of our method. We achieved superior performance compared to the existing distributed learning strategy as well as the fine-tuning only baseline. Furthermore, since the self-supervised model created using our proposed method can achieve a general semantic understanding of the image, we demonstrate its potential as a task-agnostic self-supervised foundation model for various downstream tasks, thereby expanding its applicability in the medical domain.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑