跨多数据集学习基因搜索以构建可变大小模型
Learngene Search Across Multiple Datasets for Building Variable-Sized Models
AI总结:
本文提出LSAMD方法,通过跨多数据集搜索学习基因,构建可变大小模型,减少存储和训练成本,性能与预训练微调相当。
摘要:
深度学习方法在多样化的资源约束下被广泛应用,导致模型大小各异,如Vision Transformer (ViT)系列。部署这些模型通常需要昂贵的预训练和微调。Learngene范式通过从预训练的祖先模型(Ans-Net)中提取可迁移的组件,称为学习基因,来初始化可变大小的后代模型(Des-Nets)。现有学习基因提取方法依赖单一数据集,限制了下游性能。为解决这一限制,我们提出了Learngene Search Across Multiple Datasets for Building Variable-Sized Models (LSAMD)。LSAMD将Ans-Net扩展为可搜索的超级Ans-Net,包含数据集特定的块和数据集适配器(DADs)。在训练过程中,LSAMD为每个数据集搜索最优的架构路径。在多个数据集上的实验表明,LSAMD在性能上与预训练微调方法相当,同时显著降低了存储和训练成本。
英文摘要:
Deep learning methods are widely used under diverse resource constraints, resulting in models of varying sizes, such as the Vision Transformer (ViT) series. Deploying these models typically requires costly pretraining and finetuning. The Learngene paradigm addresses this issue by extracting transferable components, called learngenes, from a pretrained ancestry model (Ans-Net) to initialize variable-sized descendant models (Des-Nets).Existing learngene extraction methods rely on a single dataset, limiting downstream performance. To address this limitation, we propose Learngene Search Across Multiple Datasets for Building Variable-Sized Models (LSAMD). LSAMD expands the Ans-Net into a searchable super Ans-Net with dataset-specific blocks and dataset adapters (DADs). During training, LSAMD searches for an optimal architecture path for each dataset. The base blocks most frequently selected across datasets are extracted as learngenes for initializing Des-Nets.Experiments on multiple datasets show that LSAMD achieves performance comparable to pretrain-finetune methods while significantly reducing storage and training costs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。