arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.08209cs.LG
收藏

跨多数据集学习基因搜索以构建可变大小模型

Learngene Search Across Multiple Datasets for Building Variable-Sized Models

School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China(教育部新一代人工智能技术及其交叉应用重点实验室(东南大学))

Boyu Shi, Junbo Zhou, Chang Liu, Xu Yang, Qiufeng Wang, Xin Geng

AI总结:

本文提出LSAMD方法,通过跨多数据集搜索学习基因,构建可变大小模型,减少存储和训练成本,性能与预训练微调相当。

摘要:

深度学习方法在多样化的资源约束下被广泛应用,导致模型大小各异,如Vision Transformer (ViT)系列。部署这些模型通常需要昂贵的预训练和微调。Learngene范式通过从预训练的祖先模型(Ans-Net)中提取可迁移的组件,称为学习基因,来初始化可变大小的后代模型(Des-Nets)。现有学习基因提取方法依赖单一数据集,限制了下游性能。为解决这一限制,我们提出了Learngene Search Across Multiple Datasets for Building Variable-Sized Models (LSAMD)。LSAMD将Ans-Net扩展为可搜索的超级Ans-Net,包含数据集特定的块和数据集适配器(DADs)。在训练过程中,LSAMD为每个数据集搜索最优的架构路径。在多个数据集上的实验表明,LSAMD在性能上与预训练微调方法相当,同时显著降低了存储和训练成本。

英文摘要:

Deep learning methods are widely used under diverse resource constraints, resulting in models of varying sizes, such as the Vision Transformer (ViT) series. Deploying these models typically requires costly pretraining and finetuning. The Learngene paradigm addresses this issue by extracting transferable components, called learngenes, from a pretrained ancestry model (Ans-Net) to initialize variable-sized descendant models (Des-Nets).Existing learngene extraction methods rely on a single dataset, limiting downstream performance. To address this limitation, we propose Learngene Search Across Multiple Datasets for Building Variable-Sized Models (LSAMD). LSAMD expands the Ans-Net into a searchable super Ans-Net with dataset-specific blocks and dataset adapters (DADs). During training, LSAMD searches for an optimal architecture path for each dataset. The base blocks most frequently selected across datasets are extracted as learngenes for initializing Des-Nets.Experiments on multiple datasets show that LSAMD achieves performance comparable to pretrain-finetune methods while significantly reducing storage and training costs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑