使用深度混合专家模型检测TempoWiC的词义偏移
Using Deep Mixture-of-Experts to Detect Word Meaning Shift for TempoWiC
摘要：
本文主要描述了TempoWiC任务的DMA提交，该任务的宏F1得分达到77.05%，在该任务中获得第一名。我们首先探讨不同预训练语言模型的影响。然后采用数据清洗、数据增强和对抗性训练策略来提高模型的泛化能力和鲁棒性。为了进一步改进，我们集成POS信息和单词语义表示使用混合专家模型（MoE）的方法。实验结果表明，MoE能很好地结合上下文、词性和词的语义特征，克服特征过度使用的问题。此外，我们使用一个模型集成的方法进行最终的预测，这已被许多研究工作证明是有效的。
英文摘要：
This paper mainly describes the dma submission to the TempoWiC task, which achieves a macro-F1 score of 77.05% and attains the first place in this task. We first explore the impact of different pre-trained language models. Then we adopt data cleaning, data augmentation, and adversarial training strategies to enhance the model generalization and robustness. For further improvement, we integrate POS information and word semantic representation using a Mixture-of-Experts (MoE) approach. The experimental results show that MoE can overcome the feature overuse issue and combine the context, POS, and word semantic features well. Additionally, we use a model ensemble method for the final prediction, which has been proven effective by many research works.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
