首页/论文/2011.06153

arXiv 2011.061532020-11-12cs.CLcs.LG

Augmenting BERT Carefully with Underrepresented Linguistic Features

Winterlight Labs；University of Toronto（多伦多大学）；Vector Institute（向量研究所）

Aparna Balagopalan, Jekaterina Novikova

备注：: Machine Learning for Health (ML4H) at NeurIPS 2020 - Extended Abstract

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

Fine-tuned Bidirectional Encoder Representations from Transformers (BERT)-based sequence classification models have proven to be effective for detecting Alzheimer's Disease (AD) from transcripts of human speech. However, previous research shows it is possible to improve BERT's performance on various tasks by augmenting the model with additional information. In this work, we use probing tasks as introspection techniques to identify linguistic information not well-represented in various layers of BERT, but important for the AD detection task. We supplement these linguistic features in which representations from BERT are found to be insufficient with hand-crafted features externally, and show that jointly fine-tuning BERT in combination with these features improves the performance of AD classification by upto 5\% over fine-tuned BERT alone.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

VGCN-BERT: Augmenting BERT with Graph Embedding for Text Classification

University of Montreal（蒙特利尔大学）

Much progress has been made recently on text classification with methods based on neural networks. In particular, models using attention mechanism such as BERT have shown to have the capability of capturing the contextual information within a sentence or document. However, their ability of capturing the global information about the vocabulary of a language is more limited. This latter is the strength of Graph Convolutional Networks (GCN). In this paper, we propose VGCN-BERT model which combines the capability of BERT with a Vocabulary Graph Convolutional Network (VGCN). Local information and g…

2020-04-12cs.CLcs.LGstat.ML

Augmenting BERT-style Models with Predictive Coding to Improve Discourse-level Representations

Pontificia Universidad Católica de Chile（智利天主教大学）；KU Leuven（鲁汶大学）；Universidad Técnica Federico Santa María（费德里科·圣玛丽亚技术大学）

Current language models are usually trained using a self-supervised scheme, where the main focus is learning representations at the word or sentence level. However, there has been limited progress in generating useful discourse-level representations. In this work, we propose to use ideas from predictive coding theory to augment BERT-style language models with a mechanism that allows them to learn suitable discourse-level representations. As a result, our proposed approach is able to predict future sentences using explicit top-down connections that operate at the intermediate layers of the netw…

2021-09-10cs.CL

在 BERT 中保留经验概率以用于小样本临床实体识别

Preserving Empirical Probabilities in BERT for Small-sample Clinical Entity Recognition

针对临床 NER 中实体标签不平衡导致少数类性能下降的问题，本文分析 BERT 在随机化数据集上的损失计算与传播机制，并提出保留经验概率的 token 分类改进方法。

2024-09-05cs.CLcs.LG

TIDE：用于评估和增强分类模型与语言模型的文本身份检测

TIDE: Textual Identity Detection for Evaluating and Augmenting Classification and Language Models

Google Research（谷歌研究院）

本文提出文本身份检测方法 TIDE 及包含 15,123 个术语的身份词典 TIDAL，通过身份标注与增强工具改善数据集和模型的公平性，实验证明其能提高标注效率、揭示更多差异并生成更公平的模型。

2024-01-12cs.CLcs.LG

Augmenting Customer Support with an NLP-based Receptionist

QuintoAndar Inc（QuintoAndar公司）；University of São Paulo（圣保罗大学）

In this paper, we show how a Portuguese BERT model can be combined with structured data in order to deploy a chatbot based on a finite state machine to create a conversational AI system that helps a real-estate company to predict its client's contact motivation. The model achieves human level results in a dataset that contains 235 unbalanced labels. Then, we also show its benefits considering the business impact comparing it against classical NLP methods.

2021-12-03cs.CL

BERT4GCN: Using BERT Intermediate Layers to Augment GCN for Aspect-based Sentiment Classification

Jinan University（暨南大学）；Guangzhou Xuanyuan Research Institute Company, Ltd.（广州轩辕研究院有限公司）；Guangzhou Yunqu Information Technology Company, Ltd.（广州云衢信息技术有限公司）

Graph-based Aspect-based Sentiment Classification (ABSC) approaches have yielded state-of-the-art results, expecially when equipped with contextual word embedding from pre-training language models (PLMs). However, they ignore sequential features of the context and have not yet made the best of PLMs. In this paper, we propose a novel model, BERT4GCN, which integrates the grammatical sequential features from the PLM of BERT, and the syntactic knowledge from dependency graphs. BERT4GCN utilizes outputs from intermediate layers of BERT and positional information between words to augment GCN (Graph…

2021-10-01cs.CL

更多相关论文收起更多论文

合成插补方法：在监督分类任务中为代表性不足类别生成最优合成文本

The Synthetic Imputation Approach: Generating Optimal Synthetic Texts For Underrepresented Categories In Supervised Classification Tasks

Purdue University（普渡大学）

本文提出合成插补方法，利用GPT-4o基于提示和五个原始示例生成合成文本，以平衡监督分类任务中代表性不足的类别，在75个原始样本时性能与完整样本相当，并有效控制过拟合。

2025-04-21cs.CL

Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representations

Paul G. Allen School of Computer Science & Engineering（保罗·G·艾伦计算机科学与工程学院）；University of Washington（华盛顿大学）

Pretrained deep contextual representations have advanced the state-of-the-art on various commonsense NLP tasks, but we lack a concrete understanding of the capability of these models. Thus, we investigate and challenge several aspects of BERT's commonsense representation abilities. First, we probe BERT's ability to classify various object attributes, demonstrating that BERT shows a strong ability in encoding various commonsense features in its embedding space, but is still deficient in many areas. Next, we show that, by augmenting BERT's pretraining data with additional data related to the def…

2019-10-04cs.CL

利用大型语言模型增强讽刺检测中的情感特征

Augmenting emotion features in irony detection with Large language modeling

University of Essex（埃塞克斯大学）

本研究提出一种结合大型语言模型与提示学习的情感文本增强方法，将微妙情感线索整合至BERT、T5和GPT-2模型中，在SemEval-2018 Task 3数据集上显著提升了讽刺检测能力。

2024-04-20cs.AIcs.CL

利用语言识别增强代码混合文本分类

Leveraging Language Identification to Enhance Code-Mixed Text Classification

SCTR’S Pune Institute of Computer Technology（SCTR’S浦那计算机技术学院）；L3Cube Pune（L3Cube浦那）；Indian Institute of Technology Madaras（印度理工学院马德拉斯分校）

针对低资源印地语-英语代码混合文本，本文提出结合词级语言识别与语言增强的BERT分类流水线，并在五个下游数据集上验证其性能提升。

2023-06-08cs.CLcs.LG

BERT偏离主题：使用体裁分类探究领域迁移挑战

BERT Goes Off-Topic: Investigating the Domain Transfer Challenge using Genre Classification

University of Strathclyde（思克莱德大学）；University of Leeds（利兹大学）

本文探究了PLMs在主题分布变化时的性能差距，通过体裁分类任务证实了BERT和GPT-3等模型在领域迁移上的挑战，并提出使用主题受控的合成文本增强训练集，使部分主题F1分数提升高达50%。

2023-11-27cs.CL

论LLM作为低资源语言标注器的局限性

On Limitations of LLM as Annotator for Low Resource Languages

Pune Institute of Computer Technology（浦那计算机技术学院）；Indian Institute of Technology Madras（印度马德拉斯理工学院）；L3Cube Labs（L3Cube实验室）

本文评估了闭源和开源LLMs作为低资源语言马拉地语标注器的表现，发现即使是GPT-4o和Llama 3.1 405B等先进模型，在分类任务上的准确率仍落后于微调BERT 10.2%和14.1%，凸显了LLM在低资源语言标注中的局限性。

2025-03-01cs.CLcs.LG

VAuLT: Augmenting the Vision-and-Language Transformer for Sentiment Classification on Social Media

University of Southern California（南加州大学）

We propose the Vision-and-Augmented-Language Transformer (VAuLT). VAuLT is an extension of the popular Vision-and-Language Transformer (ViLT), and improves performance on vision-and-language (VL) tasks that involve more complex text inputs than image captions while having minimal impact on training and inference efficiency. ViLT, importantly, enables efficient training and inference in VL tasks, achieved by encoding images using a linear projection of patches instead of an object detector. However, it is pretrained on captioning datasets, where the language input is simple, literal, and descri…

2023-01-25cs.AIcs.CLcs.CV

合成特征增强提升语言模型的泛化性能

Synthetic Feature Augmentation Improves Generalization Performance of Language Models

Mayo Clinic（梅奥诊所）

针对有限不平衡数据集下大语言模型等深度学习模型泛化差、过拟合主导类别的问题，提出嵌入空间合成特征增强方法，经多文本分类基准验证可提升不平衡场景下模型性能与鲁棒性。

2025-01-11cs.AIcs.CL

Depression detection in social media posts using affective and social norm features

National Technical University of Athens（雅典国家技术大学）；Institute for Speech and Language Processing（言语与语言处理研究所）

We propose a deep architecture for depression detection from social media posts. The proposed architecture builds upon BERT to extract language representations from social media posts and combines these representations using an attentive bidirectional GRU network. We incorporate affective information, by augmenting the text representations with features extracted from a pretrained emotion classifier. Motivated by psychological literature we propose to incorporate profanity and morality features of posts and words in our architecture using a late fusion scheme. Our analysis indicates that moral…

2023-03-24cs.CL

Emotions are Subtle: Learning Sentiment Based Text Representations Using Contrastive Learning

Carnegie Mellon University（卡内基梅隆大学）；University of California（加利福尼亚大学）

Contrastive learning techniques have been widely used in the field of computer vision as a means of augmenting datasets. In this paper, we extend the use of these contrastive learning embeddings to sentiment analysis tasks and demonstrate that fine-tuning on these embeddings provides an improvement over fine-tuning on BERT-based embeddings to achieve higher benchmarks on the task of sentiment analysis when evaluated on the DynaSent dataset. We also explore how our fine-tuned models perform on cross-domain benchmark datasets. Additionally, we explore upsampling techniques to achieve a more bala…

2021-12-02cs.CL

用于基于融合的评论项检索的自监督对比BERT微调

Self-Supervised Contrastive BERT Fine-tuning for Fusion-based Reviewed-Item Retrieval

University of Toronto（多伦多大学）；LG Electronics（LG电子）

针对评论项检索（RIR）任务缺乏标注数据的问题，提出利用自监督对比学习微调BERT嵌入，结合物品-评论两级结构和元数据进行正负样本选择，实验证明后期融合方法显著优于其他基线。

2023-08-01cs.CLcs.IRcs.LG

Information-Theoretic Probing for Linguistic Structure

University of Cambridge（剑桥大学）；Facebook AI Research（Facebook人工智能研究院）；ETH Zürich（苏黎世联邦理工学院）

The success of neural networks on a diverse set of NLP tasks has led researchers to question how much these networks actually ``know'' about natural language. Probes are a natural way of assessing this. When probing, a researcher chooses a linguistic task and trains a supervised model to predict annotations in that linguistic task from the network's learned representations. If the probe does well, the researcher may conclude that the representations encode knowledge related to the task. A commonly held belief is that using simpler models as probes is better; the logic is that simpler models wi…

2020-05-22cs.CLcs.LG