It is well known that textual data on the internet and other digital platforms contain significant levels of bias and stereotypes. Although many such texts contain stereotypes and biases that inherently exist in natural language for reasons that are not necessarily malicious, there are crucial reasons to mitigate these biases. For one, these texts are being used as training corpus to train language models for salient applications like cv-screening, search engines, and chatbots; such applications are turning out to produce discriminatory results. Also, several research findings have concluded that biased texts have significant effects on the target demographic groups. For instance, masculine-worded job advertisements tend to be less appealing to female applicants.
In this paper, we present a text style transfer model that can be used to automatically debias textual data. Our style transfer model improves on the limitations of many existing style transfer techniques such as loss of content information. Our model solves such issues by combining latent content encoding with explicit keyword replacement. We will show that this technique produces better content preservation whilst maintaining good style transfer accuracy.
Indian Institute of Technology Hyderabad（印度理工学院海得拉巴分校）
Text Style Transfer (TST) is performable through approaches such as latent space disentanglement, cycle-consistency losses, prototype editing etc. The prototype editing approach, which is known to be quite successful in TST, involves two key phases a) Masking of source style-associated tokens and b) Reconstruction of this source-style masked sentence conditioned with the target style. We follow a similar transduction method, in which we transpose the more difficult direct source to target TST task to a simpler Style-Masked Language Model (SMLM) Task, wherein, similar to BERT \cite{bert}, the g…
We propose Masker, an unsupervised text-editing method for style transfer. To tackle cases when no parallel source-target pairs are available, we train masked language models (MLMs) for both the source and the target domain. Then we find the text spans where the two models disagree the most in terms of likelihood. This allows us to identify the source tokens to delete to transform the source text to match the style of the target domain. The deleted tokens are replaced with the target MLM, and by using a padded MLM variant, we avoid having to predetermine the number of inserted tokens. Our expe…
NTT Media Intelligence Laboratories, NTT Corporation（日本电报电话公司媒体情报实验室）
This paper presents a self-supervised learning method for pointer-generator networks to improve spoken-text normalization. Spoken-text normalization that converts spoken-style text into style normalized text is becoming an important technology for improving subsequent processing such as machine translation and summarization. The most successful spoken-text normalization method to date is sequence-to-sequence (seq2seq) mapping using pointer-generator networks that possess a copy mechanism from an input sequence. However, these models require a large amount of paired data of spoken-style text an…
Research in the area of style transfer for text is currently bottlenecked by a lack of standard evaluation practices. This paper aims to alleviate this issue by experimentally identifying best practices with a Yelp sentiment dataset. We specify three aspects of interest (style transfer intensity, content preservation, and naturalness) and show how to obtain more reliable measures of them from human evaluation than in previous work. We propose a set of metrics for automated evaluation and demonstrate that they are more strongly correlated and in agreement with human judgment: direction-correcte…
On the Difference of BERT-style and CLIP-style Text Encoders
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Hong Kong University of Science and Technology（香港科技大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Baidu Inc.（百度公司）
This paper focuses on the task of sentiment transfer on non-parallel text, which modifies sentiment attributes (e.g., positive or negative) of sentences while preserving their attribute-independent content. Due to the limited capability of RNNbased encoder-decoder structure to capture deep and long-range dependencies among words, previous works can hardly generate satisfactory sentences from scratch. When humans convert the sentiment attribute of a sentence, a simple but effective approach is to only replace the original sentimental tokens in the sentence with target sentimental expressions, i…
From Detection to Mitigation: Addressing Gender Bias in Chinese Texts via Efficient Tuning and Voting-Based Rebalancing
Guangdong Provincial Key Laboratory of Quantum Engineering and Quantum Materials（广东省量子工程与量子材料重点实验室）；School of Electronic Science and Engineering (School of Microelectronics)（电子科学与工程学院（微电子学院））；South China Normal University（华南师范大学）；Guangdong Provincial Key Laboratory of Intelligent Information Processing（广东省智能信息处理重点实验室）；School of Business（商学院）；Yangzhou University（扬州大学）
South China University of Technology（华南理工大学）；HKUST（香港科技大学）；University of Illinois, Urbana Champaign（伊利诺伊大学厄巴纳-香槟分校）；Stanford University（斯坦福大学）
Solving text classification in a weakly supervised manner is important for real-world applications where human annotations are scarce. In this paper, we propose to query a masked language model with cloze style prompts to obtain supervision signals. We design a prompt which combines the document itself and "this article is talking about [MASK]." A masked language model can generate words for the [MASK] token. The generated words which summarize the content of a document can be utilized as supervision signals. We propose a latent variable model to learn a word distribution learner which associa…
Retrieval-style In-Context Learning for Few-shot Hierarchical Text Classification
Institute of Computing and Intelligence, Harbin Institute of Technology（哈尔滨工业大学计算与智能学院）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学院）
Masked Language Models are Good Heterogeneous Graph Generalizers
School of Computer Science, Beijing University of Posts and Telecommunications（北京邮电大学计算机科学学院）；Department of Computer Science, University of Illinois Chicago（伊利诺伊大学芝加哥分校计算机科学系）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Ant Group, Beijing, China（蚂蚁集团（北京））
INESC TEC（葡萄牙国家工程与系统创新研究所）；University of Porto（波尔图大学）；Cardiff University（卡迪夫大学）
Progress on commonsense reasoning is usually measured from performance improvements on Question Answering tasks designed to require commonsense knowledge. However, fine-tuning large Language Models (LMs) on these specific tasks does not directly evaluate commonsense learned during pre-training. The most direct assessments of commonsense knowledge in pre-trained LMs are arguably cloze-style tasks targeting commonsense assertions (e.g., A pen is used for [MASK].). However, this approach is restricted by the LM's vocabulary available for masked predictions, and its precision is subject to the con…
When Emotion Becomes Trigger: Emotion-style dynamic Backdoor Attack Parasitising Large Language Models
School of Cyber Science and Engineering, Sichuan University（四川大学计算机科学与工程学院）；School of Computer Science, China West Normal University（西南大学计算机科学学院）；School of Electronic and Information Engineering, Lanzhou Jiaotong University（兰州交通大学电子信息工程学院）
Beihang University（北京航空航天大学）；University of Southern California（南加州大学）
Pre-trained language models (PTLM) have achieved impressive results in a range of natural language understanding (NLU) and generation (NLG) tasks. However, current pre-training objectives such as masked token prediction (for BERT-style PTLMs) and masked span infilling (for T5-style PTLMs) do not explicitly model the relational commonsense knowledge about everyday concepts, which is crucial to many downstream tasks that need common sense to understand or generate. To augment PTLMs with concept-centric commonsense knowledge, in this paper, we propose both generative and contrastive objectives fo…
