BinaryBERT: Pushing the Limit of BERT Quantization

NeoDictaBERT：推动希伯来语BERT模型的前沿发展

NeoDictaBERT: Pushing the Frontier of BERT models for Hebrew

本文提出NeoDictaBERT系列模型，采用NeoBERT架构专门训练希伯来语，在希伯来语基准上超越现有模型，其中双语版在检索任务上表现突出，并已开源。

对 BERT 与 DistilBERT 中导致性别偏见的结构的探究

An investigation of structures responsible for gender bias in BERT and DistilBERT

本文实证探究 BERT 与 DistilBERT 中性别偏见的神经机制，发现偏见并非集中在特定层，而是由各注意力头均匀编码，且蒸馏模型中的偏见分布更均匀。

Pushing on Personality Detection from Verbal Behavior: A Transformer Meets Text Contours of Psycholinguistic Features

Research at the intersection of personality psychology, computer science, and linguistics has recently focused increasingly on modeling and predicting personality from language use. We report two major improvements in predicting personality traits from text data: (1) to our knowledge, the most comprehensive set of theory-based psycholinguistic features and (2) hybrid models that integrate a pre-trained Transformer Language Model BERT and Bidirectional Long Short-Term Memory (BLSTM) networks trained on within-text distributions ('text contours') of psycholinguistic features. We experiment with …

Multi-Task Deep Neural Networks for Natural Language Understanding

In this paper, we present a Multi-Task Deep Neural Network (MT-DNN) for learning representations across multiple natural language understanding (NLU) tasks. MT-DNN not only leverages large amounts of cross-task data, but also benefits from a regularization effect that leads to more general representations in order to adapt to new tasks and domains. MT-DNN extends the model proposed in Liu et al. (2015) by incorporating a pre-trained bidirectional transformer language model, known as BERT (Devlin et al., 2018). MT-DNN obtains new state-of-the-art results on ten NLU tasks, including SNLI, SciTai…

INGENIOUS：使用信息性数据子集进行语言模型的高效预训练

INGENIOUS: Using Informative Data Subsets for Efficient Pre-Training of Language Models

本研究提出INGENIOUS框架，利用次模优化选择训练数据中高度代表性的子集，仅用部分数据高效训练PTLMs，在保持下游性能的同时显著降低训练成本。

Transformers: "The End of History" for NLP?

Recent advances in neural architectures, such as the Transformer, coupled with the emergence of large-scale pre-trained models such as BERT, have revolutionized the field of Natural Language Processing (NLP), pushing the state of the art for a number of NLP tasks. A rich family of variations of these models has been proposed, such as RoBERTa, ALBERT, and XLNet, but fundamentally, they all remain limited in their ability to model certain kinds of information, and they cannot cope with certain information sources, which was easy for pre-existing models. Thus, here we aim to shed light on some im…