arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.05948cs.LGcs.CL
收藏

DE$^3$-BERT: 基于原型网络的距离增强型BERT早期退出

DE$^3$-BERT: Distance-Enhanced Early Exiting for BERT based on Prototypical Networks

School of Information Engineering, Tianjin University of Commerce（天津商业大学信息工程学院）；School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）；School of Artificial Intelligence and Computer Science, Nantong University（南通大学人工智能与计算机科学学院）；Faculty of Data Science, City University of Macau（澳门城市大学数据科学学院）；National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别国家重点实验室）；DataX Research Centre, School of Computing, Macquarie University（麦考瑞大学计算学院DataX研究中心）

Jianing He, Qi Zhang, Weiping Ding, Duoqian Miao, Jun Zhao, Liang Hu, Longbing Cao

备注：
16 pages
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

DE$^3$-BERT通过结合局部和全局信息提升BERT的早期退出效果，实验表明其在不同加速比例下性能更优。

摘要：

早期退出已在加速预训练语言模型（如BERT）的推理过程中表现出有效性，通过动态调整执行的层数。然而，大多数现有的早期退出方法仅考虑单个测试样本的局部信息来确定退出指标，未能利用样本群体提供的全局信息。这导致预测正确性的估计不充分，导致错误的退出决策。为弥合这一差距，我们探索了有效结合局部和全局信息的必要性，以确保推理过程中的可靠早期退出。为此，我们利用原型网络学习类别原型，并设计样本与类别原型之间的距离度量。这使我们能够利用全局信息来估计早期预测的正确性。在此基础上，我们提出了一种新的基于BERT的Distance-Enhanced Early Exiting框架（DE$^3$-BERT）。DE$^3$-BERT实现了一种混合退出策略，将经典基于熵的局部信息与基于距离的全局信息相结合，以增强预测正确性的估计，从而实现更可靠的早期退出决策。在GLUE基准上的大量实验表明，DE$^3$-BERT在不同加速比例下均优于最先进的模型，具有最小的存储或计算开销，实现了模型性能与推理效率之间的更好权衡。此外，深入分析进一步验证了我们方法的通用性和可解释性。

英文摘要：

Early exiting has demonstrated its effectiveness in accelerating the inference of pre-trained language models like BERT by dynamically adjusting the number of layers executed. However, most existing early exiting methods only consider local information from an individual test sample to determine their exiting indicators, failing to leverage the global information offered by sample population. This leads to suboptimal estimation of prediction correctness, resulting in erroneous exiting decisions. To bridge the gap, we explore the necessity of effectively combining both local and global information to ensure reliable early exiting during inference. Purposefully, we leverage prototypical networks to learn class prototypes and devise a distance metric between samples and class prototypes. This enables us to utilize global information for estimating the correctness of early predictions. On this basis, we propose a novel Distance-Enhanced Early Exiting framework for BERT (DE$^3$-BERT). DE$^3$-BERT implements a hybrid exiting strategy that supplements classic entropy-based local information with distance-based global information to enhance the estimation of prediction correctness for more reliable early exiting decisions. Extensive experiments on the GLUE benchmark demonstrate that DE$^3$-BERT consistently outperforms state-of-the-art models under different speed-up ratios with minimal storage or computational overhead, yielding a better trade-off between model performance and inference efficiency. Additionally, an in-depth analysis further validates the generality and interpretability of our method.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑