AutoDistil：用于提取大语言模型的Few-Shot任务无关神经架构搜索
AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models
摘要：
知识蒸馏（KD）方法将大模型压缩成较小的学生，其具有给定预先指定的计算成本的手动设计的学生架构。这需要几次试验来找到一个可行的学生，并进一步重复每个学生或计算预算变化的过程。我们使用神经架构搜索（NAS）自动提取几个压缩的学生与可变成本从一个大的模型。目前的工程训练一个单一的超级LM组成的数以百万计的子网的权重共享，导致不同大小的子网之间的干扰。我们的框架AutoDistil通过以下步骤解决了上述挑战：（a）将归纳偏差和分类法用于将Transformer搜索空间划分为K个紧凑的子空间（对于基本、小和微小的典型学生大小，K=3）;（b）使用任务不可知目标（例如，自我注意力蒸馏）与学生的权重共享;（c）轻量级搜索最佳学生，无需重新训练。完全与任务无关的训练和搜索允许学生在任何下游任务上进行微调。在GLUE基准测试中对最先进的KD和NAS方法进行的实验表明，AutoDistil的性能优于领先的压缩技术，计算成本降低了2.7倍，任务性能损失可以忽略不计。
英文摘要：
Knowledge distillation (KD) methods compress large models into smaller students with manually-designed student architectures given pre-specified computational cost. This requires several trials to find a viable student, and further repeating the process for each student or computational budget change. We use Neural Architecture Search (NAS) to automatically distill several compressed students with variable cost from a large model. Current works train a single SuperLM consisting of millions of subnetworks with weight-sharing, resulting in interference between subnetworks of different sizes. Our framework AutoDistil addresses above challenges with the following steps: (a) Incorporates inductive bias and heuristics to partition Transformer search space into K compact sub-spaces (K=3 for typical student sizes of base, small and tiny); (b) Trains one SuperLM for each sub-space using task-agnostic objective (e.g., self-attention distillation) with weight-sharing of students; (c) Lightweight search for the optimal student without re-training. Fully task-agnostic training and search allow students to be reused for fine-tuning on any downstream task. Experiments on GLUE benchmark against state-of-the-art KD and NAS methods demonstrate AutoDistil to outperform leading compression techniques with upto 2.7x reduction in computational cost and negligible loss in task performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
