FlexRank: 嵌套低秩知识分解用于自适应模型部署
FlexRank: Nested Low-Rank Knowledge Decomposition for Adaptive Model Deployment
AI总结：
提出FlexRank方法，通过嵌套低秩权重分解和基于重要性的整合，从预训练模型中提取不同能力的子模型，实现“一次训练，随处部署”的自适应部署。
摘要：
深度神经网络（包括大型语言模型和视觉变换器）的规模不断增长，使得从头训练成本过高，部署成本也日益增加。这些模型通常作为固定计算成本的单一整体使用，阻碍了在不同成本预算下的自适应部署。我们认为，可以从预训练模型中提取按重要性排序的嵌套组件，并在可用计算预算内选择性激活。为此，我们提出的FlexRank方法利用嵌套的、基于重要性的低秩权重分解来整合子模型，从而提取能力递增的子模型。我们的方法实现了“一次训练，随处部署”的范式，无需为每个预算从头训练即可在成本与性能之间实现优雅的权衡——推进了大型模型的实际部署。
英文摘要：
The growing scale of deep neural networks, encompassing large language models (LLMs) and vision transformers (ViTs), has made training from scratch prohibitively expensive and deployment increasingly costly. These models are often used as computational monoliths with fixed cost, hindering adaptive deployment across different cost budgets. We argue that nested components, ordered by importance, can be extracted from pretrained models and selectively activated within the available computational budget. To this end, our proposed FlexRank method leverages low-rank weight decomposition with nested, importance-based consolidation to extract submodels of increasing capabilities. Our approach enables a "train-once, deploy-everywhere" paradigm offering a graceful trade-off between cost and performance without training from scratch for each budget - advancing practical deployment of large models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
