arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2203.06390cs.CL
收藏

BiBERT:精确的完全二进制化的BERT

BiBERT: Accurate Fully Binarized BERT

Beihang University(北京航空航天大学);Baidu Inc.(百度公司);Nanyang Technological University(南洋理工大学)

Haotong Qin, Yifu Ding, Mingyuan Zhang, Qinghua Yan, Aishan Liu, Qingqing Dang, Ziwei Liu, Xianglong Liu

摘要:

大型预训练的BERT在自然语言处理(NLP)任务上取得了显着的性能,但也是计算和内存昂贵的。二值化作为一种强大的压缩方法,通过使用1位参数和逐位操作,极大地减少了计算量和内存消耗。不幸的是,BERT的完全二进制化(即,1-位权重、嵌入和激活)通常遭受显著的性能下降,并且很少有研究解决该问题。本文通过理论论证和实证分析,指出前向传播和后向传播过程中的信息退化和优化方向失配是导致性能严重下降的主要原因,并提出了一种精确的全二值化BERT--双BERT,以消除性能瓶颈。具体来说,双BERT引入了一个有效的双注意力结构,最大限度地提高代表性的信息统计和方向匹配蒸馏(DMD)计划,以优化全二进制BERT准确。大量的实验表明,BiBERT在NLP基准测试中的令人信服的裕度优于直接基线和现有的最先进的量化BERT,具有超低位激活。作为第一个完全二进制化的BERT,我们的方法产生了令人印象深刻的56.3倍和31.2倍的FLOP和模型大小的节省,展示了巨大的优势和潜力,在现实世界的资源受限的情况下,完全二进制化的BERT模型。

英文摘要:

The large pre-trained BERT has achieved remarkable performance on Natural Language Processing (NLP) tasks but is also computation and memory expensive. As one of the powerful compression approaches, binarization extremely reduces the computation and memory consumption by utilizing 1-bit parameters and bitwise operations. Unfortunately, the full binarization of BERT (i.e., 1-bit weight, embedding, and activation) usually suffer a significant performance drop, and there is rare study addressing this problem. In this paper, with the theoretical justification and empirical analysis, we identify that the severe performance drop can be mainly attributed to the information degradation and optimization direction mismatch respectively in the forward and backward propagation, and propose BiBERT, an accurate fully binarized BERT, to eliminate the performance bottlenecks. Specifically, BiBERT introduces an efficient Bi-Attention structure for maximizing representation information statistically and a Direction-Matching Distillation (DMD) scheme to optimize the full binarized BERT accurately. Extensive experiments show that BiBERT outperforms both the straightforward baseline and existing state-of-the-art quantized BERTs with ultra-low bit activations by convincing margins on the NLP benchmark. As the first fully binarized BERT, our method yields impressive 56.3 times and 31.2 times saving on FLOPs and model size, demonstrating the vast advantages and potential of the fully binarized BERT model in real-world resource-constrained scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑