轻量级竞争者:通过同伴协作与自我超越驾驭半监督文本挖掘
Lightweight Contenders: Navigating Semi-Supervised Text Mining through Peer Collaboration and Self Transcendence
AI总结:
针对轻量级模型在标签稀缺时半监督文本挖掘性能受限的问题,提出 PS-NET,融合在线蒸馏、同伴协作集成与恒定对抗扰动自我增强,并以2层蒸馏BERT取得优于FLiText和DisCo的效果。
摘要:
轻量级模型中的半监督学习(SSL)策略需要减少标注样本,并促进具有成本效益的推理。然而,训练标签稀缺所施加的模型参数约束限制了 SSL 性能。本文提出 PS-NET,这是一种专为轻量级模型的半监督文本挖掘定制的新型框架。PS-NET 结合在线蒸馏,通过模仿 Teacher 模型来训练轻量级学生模型。它还集成了一个学生同伴集成,这些同伴相互协作指导。此外,PS-NET 实施恒定对抗扰动模式,通过渐进式泛化进一步实现自我增强。我们的 PS-NET 配备了一个 2 层蒸馏 BERT,在标注数据极其稀少的 SSL 文本分类中,相比 FLiText 和 DisCo 等 SOTA 轻量级 SSL 框架表现出显著的性能提升。
英文摘要:
The semi-supervised learning (SSL) strategy in lightweight models requires reducing annotated samples and facilitating cost-effective inference. However, the constraint on model parameters, imposed by the scarcity of training labels, limits the SSL performance. In this paper, we introduce PS-NET, a novel framework tailored for semi-supervised text mining with lightweight models. PS-NET incorporates online distillation to train lightweight student models by imitating the Teacher model. It also integrates an ensemble of student peers that collaboratively instruct each other. Additionally, PS-NET implements a constant adversarial perturbation schema to further self-augmentation by progressive generalizing. Our PS-NET, equipped with a 2-layer distilled BERT, exhibits notable performance enhancements over SOTA lightweight SSL frameworks of FLiText and DisCo in SSL text classification with extremely rare labelled data.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。