预训练与自训练的比较研究
A Comparative Study of Pre-training and Self-training
AI总结:
本文通过一致设置下的系统实验,比较预训练、自训练及微调的各种组合,发现预训练加微调整体表现最佳,且自训练在半监督预训练基础上无额外增益。
摘要:
预训练和自训练是半监督学习的两种方法。预训练与自训练之间的比较已被探索过。然而,先前的工作得出了令人困惑的发现:在计算机视觉的某些任务上,自训练优于预训练;相反,在自然语言处理的某些任务上,在不可比设置的特定条件下,预训练优于自训练。我们提出一种比较性的、穷尽式的集成方法,在与数据增强可比的、一致的基础设置内,实证研究所有可行的结合预训练、自训练和微调的训练范式。我们在六个数据集、四种数据增强以及不平衡数据上,针对情感分析和自然语言推理任务进行了实验。我们的发现证实,预训练加微调的范式取得了最佳的整体性能。此外,当与半监督预训练结合时,自训练没有带来额外的好处。
英文摘要:
Pre-training and self-training are two approaches to semi-supervised learning. The comparison between pre-training and self-training has been explored. However, the previous works led to confusing findings: self-training outperforms pre-training experienced on some tasks in computer vision, and contrarily, pre-training outperforms self-training experienced on some tasks in natural language processing, under certain conditions of incomparable settings. We propose, comparatively and exhaustively, an ensemble method to empirical study all feasible training paradigms combining pre-training, self-training, and fine-tuning within consistent foundational settings comparable to data augmentation. We conduct experiments on six datasets, four data augmentation, and imbalanced data for sentiment analysis and natural language inference tasks. Our findings confirm that the pre-training and fine-tuning paradigm yields the best overall performances. Moreover, self-training offers no additional benefits when combined with semi-supervised pre-training.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。