arXiv 2210.063842022-10cs.CL

GMP*: Well-Tuned Gradual Magnitude Pruning Can Outperform Most BERT-Pruning Methods

Institute of Science and Technology Austria（奥地利科学技术研究所）；Neural Magic Inc.（Neural Magic公司）

Eldar Kurtic, Dan Alistarh

英文摘要：

We revisit the performance of the classic gradual magnitude pruning (GMP) baseline for large language models, focusing on the classic BERT benchmark on various popular tasks. Despite existing evidence in the literature that GMP performs poorly, we show that a simple and general variant, which we call GMP*, can match and sometimes outperform more complex state-of-the-art methods. Our results provide a simple yet strong baseline for future work, highlight the importance of parameter tuning for baselines, and even improve the performance of the state-of-the-art second-order pruning method in this setting.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Progressively Stacking 2.0: A Multi-stage Layerwise Training Method for BERT Training Speedup

Alibaba Group（阿里巴巴集团）

Pre-trained language models, such as BERT, have achieved significant accuracy gain in many natural language processing tasks. Despite its effectiveness, the huge number of parameters makes training a BERT model computationally very challenging. In this paper, we propose an efficient multi-stage layerwise training (MSLT) approach to reduce the training time of BERT. We decompose the whole training process into several stages. The training is started from a small model with only a few encoder layers and we gradually increase the depth of the model by adding new encoder layers. At each stage, we …

2020-11-27cs.CL

Sparse*BERT：稀疏模型泛化到新任务和新领域

Sparse*BERT: Sparse Models Generalize To New tasks and Domains

the University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Neural Magic Inc.（Neural Magic公司）

本文研究 Gradual Unstructured Magnitude Pruning 剪枝模型的跨领域与任务迁移，证明通用稀疏模型 Sparse*BERT 可通过预训练转化为 SparseBioBERT，且仅用 10% 参数即可匹配 BioBERT 质量。

2023-04-05cs.AIcs.CL

理解面向 DNA 基础模型的类 BERT 预训练

Toward Understanding BERT-Like Pre-Training for DNA Foundation Models

School of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学计算机科学与技术学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Guangzhou National Laboratory（广州国家实验室）

本文针对 DNA 基础模型直接套用 BERT 预训练的问题开展实证研究，提出通过扩展掩码边界逐步增加难度的 RandomMask，并在 6 个下游任务上达到 SOTA。

2024-09-08cs.AIcs.CLcs.LG

ERNIE-Tiny : A Progressive Distillation Framework for Pretrained Transformer Compression

Baidu Inc.（百度公司）

Pretrained language models (PLMs) such as BERT adopt a training paradigm which first pretrain the model in general data and then finetune the model on task-specific data, and have recently achieved great success. However, PLMs are notorious for their enormous parameters and hard to be deployed on real-life applications. Knowledge distillation has been prevailing to address this problem by transferring knowledge from a large teacher to a much smaller student over a set of data. We argue that the selection of thee three key components, namely teacher, training data, and learning objective, is cr…

2021-06-04cs.CL

Dynamic Language Models for Continuously Evolving Content

Google Research（谷歌研究院）

The content on the web is in a constant state of flux. New entities, issues, and ideas continuously emerge, while the semantics of the existing conversation topics gradually shift. In recent years, pre-trained language models like BERT greatly improved the state-of-the-art for a large spectrum of content understanding tasks. Therefore, in this paper, we aim to study how these language models can be adapted to better handle continuously evolving web content. In our study, we first analyze the evolution of 2013 - 2019 Twitter data, and unequivocally confirm that a BERT model trained on past twee…

2021-06-11cs.CLcs.LG