一切尽在[MASK]中：简单的指令微调使类BERT掩码语言模型成为生成式分类器
It's All in The [MASK]: Simple Instruction-Tuning Enables BERT-like Masked Language Models As Generative Classifiers
AI总结：
本文提出ModernBERT-Large-Instruct模型，通过简单指令微调利用MLM头实现生成式分类，零样本性能优于同类LLM，微调后匹配或超越传统分类头方法，为仅编码器模型在下游任务的应用提供新方向。
摘要：
尽管BERT和ModernBERT等仅编码器模型在现实世界的自然语言处理应用中无处不在，但与基于解码器的大型语言模型（LLM）相比，它们对特定任务分类头的传统依赖限制了其适用性。在这项工作中，我们引入了ModernBERT-Large-Instruct，这是一个0.4B参数的编码器模型，它利用其掩码语言建模（MLM）头进行生成式分类。我们的方法采用了一个刻意简化的训练循环和推理机制，不需要繁重的预处理、精心设计的提示或架构修改。ModernBERT-Large-Instruct在分类和基于知识的任务上都表现出强大的零样本性能，在MMLU上优于类似规模的LLM，并以少60%的参数达到了Llama3-1B在MMLU上性能的93%。我们还证明，当进行微调时，使用MLM头的生成式方法在各种自然语言理解（NLU）任务中匹配甚至超过了传统的分类头方法。这种能力特别出现在使用当代多样化数据混合训练的模型中，而使用低容量、多样性较差的数据训练的模型性能明显较弱。尽管是初步的，这些结果证明了在下游任务中使用原始的生成式掩码语言建模头而非传统特定任务头的潜力。我们的工作表明，进一步探索这一领域是有必要的，强调了未来改进的许多途径。
英文摘要：
While encoder-only models such as BERT and ModernBERT are ubiquitous in real-world NLP applications, their conventional reliance on task-specific classification heads can limit their applicability compared to decoder-based large language models (LLMs). In this work, we introduce ModernBERT-Large-Instruct, a 0.4B-parameter encoder model that leverages its masked language modelling (MLM) head for generative classification. Our approach employs an intentionally simple training loop and inference mechanism that requires no heavy pre-processing, heavily engineered prompting, or architectural modifications. ModernBERT-Large-Instruct exhibits strong zero-shot performance on both classification and knowledge-based tasks, outperforming similarly sized LLMs on MMLU and achieving 93% of Llama3-1B's MMLU performance with 60% less parameters. We also demonstrate that, when fine-tuned, the generative approach using the MLM head matches or even surpasses traditional classification-head methods across diverse NLU tasks.This capability emerges specifically in models trained on contemporary, diverse data mixes, with models trained on lower volume, less-diverse data yielding considerably weaker performance. Although preliminary, these results demonstrate the potential of using the original generative masked language modelling head over traditional task-specific heads for downstream tasks. Our work suggests that further exploration into this area is warranted, highlighting many avenues for future improvements.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
