arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.09374cs.CLcs.LG
收藏

Shuffle & Divide:面向长文本的对比学习

Shuffle & Divide: Contrastive Learning for Long Text

Samsung SDS(三星SDS)

Joonseok Lee, Seongho Joe, Kyoungwon Park, Bogun Kim, Hoyoung Kang, Jaeseon Park, Youngjune Gwon

AI总结:

本文提出基于 Shuffle and Divide 增强与对比学习的长文本自监督方法,通过重复对比更新和聚类实现无监督分类,并在多个数据集上显著提升性能。

摘要:

我们提出了一种基于对比学习的长文本文档自监督学习方法。该方法的关键是 Shuffle and Divide(SaD),这是一种简单的文本增强算法,用于设置基于 BERT 的文档嵌入进行对比更新所需的预训练任务。SaD 将一个文档拆分为两个子文档,这两个子文档包含整个文档中经过随机打乱的词语。这些子文档被视为正样本,而语料库中的所有其他文档则作为负样本。在 SaD 之后,我们重复对比更新和聚类阶段,直到收敛。为文本文档打标签天然是一项耗时且繁琐的任务,而我们的方法能够帮助减少人力投入;人力是 AI 中最昂贵的资源。我们通过在 20 Newsgroups、Reuters-21578、BBC 和 BBCSport 数据集上执行无监督文本分类,对该方法进行了实证评估。特别是,我们的方法在 20 Newsgroups 上将当前最先进方法 SS-SB-MT 的准确率提升了 20.94%。我们还在 Reuters-21578 上取得了最先进性能,并在 BBC 和 BBCSport 数据集的无监督分类中获得了极高的准确率(超过 95%)。

英文摘要:

We propose a self-supervised learning method for long text documents based on contrastive learning. A key to our method is Shuffle and Divide (SaD), a simple text augmentation algorithm that sets up a pretext task required for contrastive updates to BERT-based document embedding. SaD splits a document into two sub-documents containing randomly shuffled words in the entire documents. The sub-documents are considered positive examples, leaving all other documents in the corpus as negatives. After SaD, we repeat the contrastive update and clustering phases until convergence. It is naturally a time-consuming, cumbersome task to label text documents, and our method can help alleviate human efforts, which are most expensive resources in AI. We have empirically evaluated our method by performing unsupervised text classification on the 20 Newsgroups, Reuters-21578, BBC, and BBCSport datasets. In particular, our method pushes the current state-of-the-art, SS-SB-MT, on 20 Newsgroups by 20.94% in accuracy. We also achieve the state-of-the-art performance on Reuters-21578 and exceptionally-high accuracy performances (over 95%) for unsupervised classification on the BBC and BBCSport datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑