arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.20144cs.CLcs.AIcs.LG
收藏

EELBERT:通过动态嵌入实现微型模型

EELBERT: Tiny Models through Dynamic Embeddings

Apple(苹果公司)

Gabrielle Cohn, Rishika Agarwal, Deepanshu Gupta, Siddharth Patwardhan

AI总结:

EELBERT通过用动态嵌入计算替换输入嵌入层来压缩BERT模型,在GLUE基准上以极小性能损失实现显著体积缩减,其中UNO-EELBERT仅1.2 MB且得分接近BERT-tiny。

摘要:

我们提出了EELBERT,这是一种用于压缩基于Transformer的模型(例如BERT)的方法,对下游任务准确性的影响极小。这是通过将模型的输入嵌入层替换为动态的、即在运行中进行的嵌入计算来实现的。由于输入嵌入层在模型大小中占有很大比例,尤其是对于较小的BERT变体而言,用嵌入计算函数替换该层有助于我们显著减小模型大小。在GLUE基准上的实证评估表明,与传统的BERT模型相比,我们的BERT变体(EELBERT)仅出现极小的性能回退。通过这种方法,我们能够开发出最小的模型UNO-EELBERT,其GLUE得分与完全训练的BERT-tiny相差在4%以内,同时体积小15倍(1.2 MB)。

英文摘要:

We introduce EELBERT, an approach for compression of transformer-based models (e.g., BERT), with minimal impact on the accuracy of downstream tasks. This is achieved by replacing the input embedding layer of the model with dynamic, i.e. on-the-fly, embedding computations. Since the input embedding layer accounts for a significant fraction of the model size, especially for the smaller BERT variants, replacing this layer with an embedding computation function helps us reduce the model size significantly. Empirical evaluation on the GLUE benchmark shows that our BERT variants (EELBERT) suffer minimal regression compared to the traditional BERT models. Through this approach, we are able to develop our smallest model UNO-EELBERT, which achieves a GLUE score within 4% of fully trained BERT-tiny, while being 15x smaller (1.2 MB) in size.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑