ESCOXLM-R:面向就业市场领域的多语言分类体系驱动预训练
ESCOXLM-R: Multilingual Taxonomy-driven Pre-training for the Job Market Domain
AI总结:
提出基于XLM-R的多语言模型ESCOXLM-R,利用ESCO分类体系进行领域自适应预训练,在就业市场多项任务的多数数据集上达到SOTA。
摘要:
计算就业市场领域中自然语言处理(NLP)任务基准的不断增加,凸显了对能够处理技能抽取、技能分类、职位分类和去标识化等就业相关任务的方法的需求。尽管已经开发了一些针对就业市场领域的方法,但仍缺乏用于这些任务的通用多语言模型和基准。在本研究中,我们引入了一种名为ESCOXLM-R的语言模型,该模型基于XLM-R,利用欧洲技能、能力、资格和职业(ESCO)分类体系进行领域自适应预训练,涵盖27种语言。ESCOXLM-R的预训练目标包括动态掩码语言建模,以及一个用于引导多语言ESCO分类关系的新颖附加目标。我们在4种语言的6个序列标注和3个分类任务上全面评估了ESCOXLM-R的性能,发现它在9个数据集中的6个上取得了最先进的结果。我们的分析表明,ESCOXLM-R在短跨度上表现更好,并且在实体级和表面级跨度F1上优于XLM-R,这可能是由于ESCO包含较短的技能和职业名称,并且编码了实体级信息。
英文摘要:
The increasing number of benchmarks for Natural Language Processing (NLP) tasks in the computational job market domain highlights the demand for methods that can handle job-related tasks such as skill extraction, skill classification, job title classification, and de-identification. While some approaches have been developed that are specific to the job market domain, there is a lack of generalized, multilingual models and benchmarks for these tasks. In this study, we introduce a language model called ESCOXLM-R, based on XLM-R, which uses domain-adaptive pre-training on the European Skills, Competences, Qualifications and Occupations (ESCO) taxonomy, covering 27 languages. The pre-training objectives for ESCOXLM-R include dynamic masked language modeling and a novel additional objective for inducing multilingual taxonomical ESCO relations. We comprehensively evaluate the performance of ESCOXLM-R on 6 sequence labeling and 3 classification tasks in 4 languages and find that it achieves state-of-the-art results on 6 out of 9 datasets. Our analysis reveals that ESCOXLM-R performs better on short spans and outperforms XLM-R on entity-level and surface-level span-F1, likely due to ESCO containing short skill and occupation titles, and encoding information on the entity-level.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。