本文提出基于 Shuffle and Divide 增强与对比学习的长文本自监督方法,通过重复对比更新和聚类实现无监督分类,并在多个数据集上显著提升性能。
摘要:
我们提出了一种基于对比学习的长文本文档自监督学习方法。该方法的关键是 Shuffle and Divide(SaD),这是一种简单的文本增强算法,用于设置基于 BERT 的文档嵌入进行对比更新所需的预训练任务。SaD 将一个文档拆分为两个子文档,这两个子文档包含整个文档中经过随机打乱的词语。这些子文档被视为正样本,而语料库中的所有其他文档则作为负样本。在 SaD 之后,我们重复对比更新和聚类阶段,直到收敛。为文本文档打标签天然是一项耗时且繁琐的任务,而我们的方法能够帮助减少人力投入;人力是 AI 中最昂贵的资源。我们通过在 20 Newsgroups、Reuters-21578、BBC 和 BBCSport 数据集上执行无监督文本分类,对该方法进行了实证评估。特别是,我们的方法在 20 Newsgroups 上将当前最先进方法 SS-SB-MT 的准确率提升了 20.94%。我们还在 Reuters-21578 上取得了最先进性能,并在 BBC 和 BBCSport 数据集的无监督分类中获得了极高的准确率(超过 95%)。
英文摘要:
We propose a self-supervised learning method for long text documents based on contrastive learning. A key to our method is Shuffle and Divide (SaD), a simple text augmentation algorithm that sets up a pretext task required for contrastive updates to BERT-based document embedding. SaD splits a document into two sub-documents containing randomly shuffled words in the entire documents. The sub-documents are considered positive examples, leaving all other documents in the corpus as negatives. After SaD, we repeat the contrastive update and clustering phases until convergence. It is naturally a time-consuming, cumbersome task to label text documents, and our method can help alleviate human efforts, which are most expensive resources in AI. We have empirically evaluated our method by performing unsupervised text classification on the 20 Newsgroups, Reuters-21578, BBC, and BBCSport datasets. In particular, our method pushes the current state-of-the-art, SS-SB-MT, on 20 Newsgroups by 20.94% in accuracy. We also achieve the state-of-the-art performance on Reuters-21578 and exceptionally-high accuracy performances (over 95%) for unsupervised classification on the BBC and BBCSport datasets.
On the duality between contrastive and non-contrastive self-supervised learning
Meta AI - FAIR(Meta人工智能实验室(基础人工智能研究部));Univ Gustave Eiffel(古斯塔夫·埃菲尔大学);CNRS(法国国家科学研究中心);New York University(纽约大学);Inria(法国国家信息与自动化研究所);École normale supérieure(巴黎高等师范学院);PSL Research University(巴黎文理研究大学)
Divide and Contrast: Learning Robust Temporal Features without Augmentation
Department of Computer Science, Norwegian University of Science(挪威科学与技术大学计算机科学系);Department of Computer Science, United States Naval Academy(美国海军学院计算机科学系)
ChemCLIP: Bridging Organic and Inorganic Anticancer Compounds Through Contrastive Learning
Li Ka Shing Faculty of Medicine, The University of Hong Kong(香港大学李嘉诚医学院);Department of Diagnostic Radiology, Li Ka Shing Faculty of Medicine, The University of Hong Kong(香港大学李嘉诚医学院诊断放射学系);Department of Chemistry, The University of Hong Kong(香港大学化学系)
Scale-specific auxiliary multi-task contrastive learning for deep liver vessel segmentation
LaTIM UMR 1101(拉蒂姆UMR 1101机构);Inserm(法国国家健康与医学研究院);University of Western Brittany(西布列塔尼大学);University Hospital of Brest(布雷斯特大学医院);University Hospital of Poitiers(普瓦捷大学医院);ICube UMR 7357(ICube UMR 7357机构);CNRS(法国国家科学研究中心);IMT Atlantique(大西洋电信学院)
Label Informed Contrastive Pretraining for Node Importance Estimation on Knowledge Graphs
Tsinghua University(清华大学);Fuyao Institute of Technology(福耀科技大学);Chinese Academy of Sciences(中国科学院);Southern University of Science and Technology(南方科技大学);Fuzhou Institute of Data Technology(福州数据技术研究院)