利用语用掩蔽和替代语微调提高社会意义检测
Improving Social Meaning Detection with Pragmatic Masking and Surrogate Fine-Tuning
摘要：
掩蔽的语言模型（MLM）是用与下游微调的目标不匹配的去噪目标进行预训练的。我们提出了语用掩蔽和代理微调作为两个互补的策略，利用社会线索，以推动预先训练的表示对广泛的一类社会意义的任务有用的概念。我们在$15$不同的Twitter数据集上测试我们的模型，以进行社会意义检测。我们的方法在竞争基线上实现了$2.34\%$ $F_1$，同时优于在大型数据集上预训练的特定领域语言模型。我们的方法在少样本学习中也表现出色：只有训练数据的$5\%$（严重的Few-Shot），我们的方法能够实现令人印象深刻的$68.54\%$平均$F_1$。这些方法也是语言不可知的，正如我们在涉及来自三种不同语言的六个数据集的zero-shot设置中所示。
英文摘要：
Masked language models (MLMs) are pre-trained with a denoising objective that is in a mismatch with the objective of downstream fine-tuning. We propose pragmatic masking and surrogate fine-tuning as two complementing strategies that exploit social cues to drive pre-trained representations toward a broad set of concepts useful for a wide class of social meaning tasks. We test our models on $15$ different Twitter datasets for social meaning detection. Our methods achieve $2.34\%$ $F_1$ over a competitive baseline, while outperforming domain-specific language models pre-trained on large datasets. Our methods also excel in few-shot learning: with only $5\%$ of training data (severely few-shot), our methods enable an impressive $68.54\%$ average $F_1$. The methods are also language agnostic, as we show in a zero-shot setting involving six datasets from three different languages.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
