Train No Evil: Selective Masking for Task-Guided Pre-Training
Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Institute for Artificial Intelligence, Tsinghua University（清华大学人工智能研究院）；State Key Lab on Intelligent Technology and Systems, Tsinghua University（清华大学智能技术与系统国家重点实验室）
Yuxian Gu, Zhengyan Zhang, Xiaozhi Wang, Zhiyuan Liu, Maosong Sun
Recently, pre-trained language models mostly follow the pre-train-then-fine-tuning paradigm and have achieved great performance on various downstream tasks. However, since the pre-training stage is typically task-agnostic and the fine-tuning stage usually suffers from insufficient supervised data, the models cannot always well capture the domain-specific and task-specific patterns. In this paper, we propose a three-stage framework by adding a task-guided pre-training stage with selective masking between general pre-training and fine-tuning. In this stage, the model is trained by masked language modeling on in-domain unsupervised data to learn domain-specific patterns and we propose a novel selective masking strategy to learn task-specific patterns. Specifically, we design a method to measure the importance of each token in sequences and selectively mask the important tokens. Experimental results on two sentiment analysis tasks show that our method can achieve comparable or even better performance with less than 50% of computation cost, which indicates our method is both effective and efficient. The source code of this paper can be obtained from https://github.com/thunlp/SelectiveMasking.
DHI: Leveraging Diverse Hallucination Induction for Enhanced Contrastive Factuality Control in Large Language Models
School of Computer Science, Wuhan University（武汉大学计算机学院）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）
We present an efficient method of utilizing pretrained language models, where we learn selective binary masks for pretrained weights in lieu of modifying them through finetuning. Extensive evaluations of masking BERT and RoBERTa on a series of NLP tasks show that our masking scheme yields performance comparable to finetuning, yet has a much smaller memory footprint when several tasks need to be inferred simultaneously. Through intrinsic evaluations, we show that representations computed by masked language models encode information necessary for solving downstream tasks. Analyzing the loss land…
Pre-training a language model and then fine-tuning it for downstream tasks has demonstrated state-of-the-art results for various NLP tasks. Pre-training is usually independent of the downstream task, and previous works have shown that this pre-training alone might not be sufficient to capture the task-specific nuances. We propose a way to tailor a pre-trained BERT model for the downstream task via task-specific masking before the standard supervised fine-tuning. For this, a word list is first collected specific to the task. For example, if the task is sentiment classification, we collect a sma…
Defensive Dual Masking for Robust Adversarial Defense
School of Computing and Information Technology, University of Wollongong（伍伦贡大学计算与信息技术学院）；School of Computer, Data and Mathematical Sciences, Western Sydney University（西悉尼大学计算机、数据与数学学院）；NVIDIA（英伟达公司）
TreeDiff: AST-Guided Code Generation with Diffusion LLMs
University of Connecticut（康涅狄格大学）；San Francisco State University（旧金山州立大学）；University of California, Riverside（加州大学河滨分校）；National University of Singapore（新加坡国立大学）；University of Pittsburgh（匹兹堡大学）；Florida State University（佛罗里达州立大学）
Microsoft Azure AI（微软Azure人工智能部门）；Shanghai University of Finance and Economics（上海财经大学）
We introduce GENIUS: a conditional text generation model using sketches as input, which can fill in the missing contexts for a given sketch (key information consisting of textual spans, phrases, or words, concatenated by mask tokens). GENIUS is pre-trained on a large-scale textual corpus with a novel reconstruction from sketch objective using an extreme and selective masking strategy, enabling it to generate diverse and high-quality texts given sketches. Comparison with other competitive conditional language models (CLMs) reveals the superiority of GENIUS's text generation quality. We further …
Manipal Institute of Technology（马尼帕尔理工学院）；Manipal Academy of Higher Education（马尼帕尔高等教育学院）
Social media has become a platform for people to stand up and raise their voices against social and criminal acts. Vocalization of such information has allowed the investigation and identification of criminals. However, revealing such sensitive information may jeopardize the victim's safety. We propose #maskUp, a safe method for information communication in a secure fashion to the relevant authorities, discouraging potential bullying of the victim. This would ensure security by conserving their privacy through natural language processing supplemented with selective encryption for sensitive att…
Robust Biomedical Publication Type and Study Design Classification with Knowledge-Guided Perturbations
School of Information Sciences（信息科学学院）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Department of Psychiatry（精神病学系）；University of Illinois Chicago（伊利诺伊大学芝加哥分校）
