PYRA：并行让渡再激活方法用于训练-推理高效的任务适配
PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptation
AI总结：
针对大规模Transformer任务适配中训练与推理效率难以兼顾的问题，提出并行让渡再激活（PYRA）方法，通过自适应权重感知数据分布并再激活令牌调制，实验证明其在低高压缩率下均优于现有方法。
摘要：
近年来，Transformer的规模迅速增长，这在任务适配范围内引入了关于训练开销和推理效率的巨大挑战。现有工作，即参数高效微调（PEFT）和模型压缩，分别研究了这些挑战。然而，PEFT无法保证原始骨干网络的推理效率，特别是对于大规模模型。模型压缩需要大量的训练成本用于结构搜索和重新训练。因此，它们的简单组合无法保证以最小成本同时实现训练效率和推理效率。在本文中，我们提出了一种新颖的并行让渡再激活（PYRA）方法，以应对这种训练-推理高效任务适配的挑战。PYRA首先利用并行让渡自适应权重来全面感知下游任务中的数据分布。然后，对要合并的令牌应用一种用于令牌调制的再激活策略，从而产生校准的令牌特征。大量实验表明，PYRA在低压缩率和高压缩率下均优于所有竞争方法，展示了其在大规模基础模型中保持训练效率和推理效率的有效性和优越性。我们的代码可在https://github.com/THU-MIG/PYRA获取。
英文摘要：
Recently, the scale of transformers has grown rapidly, which introduces considerable challenges in terms of training overhead and inference efficiency in the scope of task adaptation. Existing works, namely Parameter-Efficient Fine-Tuning (PEFT) and model compression, have separately investigated the challenges. However, PEFT cannot guarantee the inference efficiency of the original backbone, especially for large-scale models. Model compression requires significant training costs for structure searching and re-training. Consequently, a simple combination of them cannot guarantee accomplishing both training efficiency and inference efficiency with minimal costs. In this paper, we propose a novel Parallel Yielding Re-Activation (PYRA) method for such a challenge of training-inference efficient task adaptation. PYRA first utilizes parallel yielding adaptive weights to comprehensively perceive the data distribution in downstream tasks. A re-activation strategy for token modulation is then applied for tokens to be merged, leading to calibrated token features. Extensive experiments demonstrate that PYRA outperforms all competing methods under both low compression rate and high compression rate, demonstrating its effectiveness and superiority in maintaining both training efficiency and inference efficiency for large-scale foundation models. Our code is available at https://github.com/THU-MIG/PYRA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
