Getting ViT in Shape:面向计算最优模型设计的缩放定律
Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design
AI总结:
本文改进缩放定律以推断视觉Transformer的计算最优模型形状(宽度和深度),提出SoViT,在同等计算量下以不到一半的推理成本取得与两倍规模模型相当的性能。
摘要:
缩放定律最近被用于推导给定计算时长下的计算最优模型大小(参数量)。我们推进并改进了此类方法,以推断计算最优的模型形状(如宽度和深度),并在视觉Transformer中成功实现了这一点。我们的形状优化视觉Transformer(SoViT)取得了与两倍于其规模的模型相竞争的结果,尽管其预训练使用的计算量相当。例如,SoViT-400m/14在ILSRCV2012上达到90.3%的微调精度,超越了更大的ViT-g/14,并在相同设置下逼近ViT-G/14,同时推理成本不到后者的一半。我们在图像分类、图像描述、VQA和零样本迁移等多个任务上进行了全面评估,证明了模型在广泛领域的有效性并指出其局限性。总体而言,我们的发现挑战了盲目扩大视觉模型的流行做法,为更理性的缩放铺平了道路。
英文摘要:
Scaling laws have been recently employed to derive compute-optimal model size (number of parameters) for a given compute duration. We advance and refine such methods to infer compute-optimal model shapes, such as width and depth, and successfully implement this in vision transformers. Our shape-optimized vision transformer, SoViT, achieves results competitive with models that exceed twice its size, despite being pre-trained with an equivalent amount of compute. For example, SoViT-400m/14 achieves 90.3% fine-tuning accuracy on ILSRCV2012, surpassing the much larger ViT-g/14 and approaching ViT-G/14 under identical settings, with also less than half the inference cost. We conduct a thorough evaluation across multiple tasks, such as image classification, captioning, VQA and zero-shot transfer, demonstrating the effectiveness of our model across a broad range of domains and identifying limitations. Overall, our findings challenge the prevailing approach of blindly scaling up vision models and pave a path for a more informed scaling.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。