The automatic identification of propaganda has gained significance in recent years due to technological and social changes in the way news is generated and consumed. That this task can be addressed effectively using BERT, a powerful new architecture which can be fine-tuned for text classification tasks, is not surprising. However, propaganda detection, like other tasks that deal with news documents and other forms of decontextualized social communication (e.g. sentiment analysis), inherently deals with data whose categories are simultaneously imbalanced and dissimilar. We show that BERT, while capable of handling imbalanced classes with no additional data augmentation, does not generalise well when the training and test data are sufficiently dissimilar (as is often the case with news sources, whose topics evolve over time). We show how to address this problem by providing a statistical measure of similarity between datasets and a method of incorporating cost-weighting into BERT when the training and test sets are dissimilar. We test these methods on the Propaganda Techniques Corpus (PTC) and achieve the second-highest score on sentence-level propaganda classification.
Continual or lifelong learning has been a long-standing challenge in machine learning to date, especially in natural language processing (NLP). Although state-of-the-art language models such as BERT have ushered in a new era in this field due to their outstanding performance in multitask learning scenarios, they suffer from forgetting when being exposed to a continuous stream of data with shifting data distributions. In this paper, we introduce DRILL, a novel continual learning architecture for open-domain text classification. DRILL leverages a biologically inspired self-organizing neural arch…
Covid-19 has spread across the world and several vaccines have been developed to counter its surge. To identify the correct sentiments associated with the vaccines from social media posts, we fine-tune various state-of-the-art pre-trained transformer models on tweets associated with Covid-19 vaccines. Specifically, we use the recently introduced state-of-the-art pre-trained transformer models RoBERTa, XLNet and BERT, and the domain-specific transformer models CT-BERT and BERTweet that are pre-trained on Covid-19 tweets. We further explore the option of text augmentation by oversampling using L…
A Unified BERT-CNN-BiLSTM Framework for Simultaneous Headline Classification and Sentiment Analysis of Bangla News
Department of CCE, International Islamic University Chittagong（国际伊斯兰大学丘拉图加冈分校计算机工程系）；Department of ICE, Noakhali Science and Technology University（诺阿克利科学与技术大学信息工程系）；Department of CSE, Banladesh University of Engineering and Technology（孟加拉工程与技术大学计算机科学与工程系）；Dept. of CSE, IIUC, Chattogram, Bangladesh（国际伊斯兰大学丘拉图加冈分校计算机科学与工程系）；School of CSE, University of Southern Mississippi（密西西比大学南方分校计算机科学学院）；Department of CSE, Netrokona University（纳特罗克纳大学计算机科学与工程系）；mPower Social Enterprise（mPower社会企业）
NLP-LTU at SemEval-2023 Task 10: The Impact of Data Augmentation and Semi-Supervised Learning Techniques on Text Classification Performance on an Imbalanced Dataset
Much research has been done for debunking and analysing fake news. Many researchers study fake news detection in the last year, but many are limited to social media data. Currently, multiples fact-checkers are publishing their results in various formats. Also, multiple fact-checkers use different labels for the fake news, making it difficult to make a generalisable classifier. With the merge classes, the performance of the machine model can be enhanced. This domain categorisation will help group the article, which will help save the manual effort in assigning the claim verification. In this pa…
Classification of Radiological Text in Small and Imbalanced Datasets in a Non-English Language
Rigshospitalet（哥本哈根大学医院（ Rigshospitalet））；Medical University of Innsbruck（因斯布鲁克医科大学）；Technical University of Denmark（丹麦技术大学）；University of Copenhagen（哥本哈根大学）
Many pairwise classification tasks, such as paraphrase detection and open-domain question answering, naturally have extreme label imbalance (e.g., $99.99\%$ of examples are negatives). In contrast, many recent datasets heuristically choose examples to ensure label balance. We show that these heuristics lead to trained models that generalize poorly: State-of-the art models trained on QQP and WikiQA each have only $2.4\%$ average precision when evaluated on realistically imbalanced test data. We instead collect training data with active learning, using a BERT-based embedding model to efficiently…
Small and imbalanced datasets commonly seen in healthcare represent a challenge when training classifiers based on deep learning models. So motivated, we propose a novel framework based on BioBERT (Bidirectional Encoder Representations from Transformers forBiomedical TextMining). Specifically, (i) we introduce Label Embeddings for Self-Attention in each layer of BERT, which we call LESA-BERT, and (ii) by distilling LESA-BERT to smaller variants, we aim to reduce overfitting and model size when working on small datasets. As an application, our framework is utilized to build a model for patient …
A Knowledge-Informed Large Language Model Framework for U.S. Nuclear Power Plant Shutdown Initiating Event Classification for Probabilistic Risk Assessment
University of Idaho（爱达荷大学）；Idaho National Laboratory（爱达荷国家实验室）
Multi-Level Attention and Contrastive Learning for Enhanced Text Classification with an Optimized Transformer
Stevens Institute of Technology（史蒂文斯理工学院）；San Francisco State University（旧金山州立大学）；University of Minnesota（明尼苏达大学）；The Chinese University of Hong Kong（香港中文大学）；New York University（纽约大学）
Propaganda spreads the ideology and beliefs of like-minded people, brainwashing their audiences, and sometimes leading to violence. SemEval 2020 Task-11 aims to design automated systems for news propaganda detection. Task-11 consists of two sub-tasks, namely, Span Identification - given any news article, the system tags those specific fragments which contain at least one propaganda technique; and Technique Classification - correctly classify a given propagandist statement amongst 14 propaganda techniques. For sub-task 1, we use contextual embeddings extracted from pre-trained transformer model…
Consumers often read product reviews to inform their buying decision, as some consumers want to know a specific component of a product. However, because typical sentences on product reviews contain various details, users must identify sentences about components they want to know amongst the many reviews. Therefore, we aimed to develop a system that identifies and collects component and aspect information of products in sentences. Our BERT-based classifiers assign labels referring to components and aspects to sentences in reviews and extract sentences with comments on specific components and as…
Large Language Multimodal Models for 5-Year Chronic Disease Cohort Prediction Using EHR Data
School of Systems and Enterprises, Stevens Institute of Technology（史蒂文斯理工学院系统与企业学院）；National Yang Ming Chiao Tung University（国立阳明交通大学）；Institute of Hospital and Health Care Administration, National Yang Ming Chiao Tung University（国立阳明交通大学医院暨健康保健管理研究所）；School of Computing and Information Science, Florida International University（佛罗里达国际大学计算与信息科学学院）；University of Michigan（密歇根大学）；Far Eastern Memorial Hospital（远东纪念医院）
