使多语言嵌入模型适配历史卢森堡语
Adapting Multilingual Embedding Models to Historical Luxembourgish
AI总结：
针对历史卢森堡语跨语言语义搜索，本文构建平行语料与Historical LB Bitext Mining评估集，并通过对比学习或知识蒸馏适配多语言嵌入模型，显著提升准确率。
摘要：
日益增长的数字化历史文本数量要求利用文本嵌入实现有效的语义搜索。然而，预训练多语言模型因OCR噪声和过时拼写，在处理历史内容时面临挑战。本研究考察了用于低资源语言——历史卢森堡语（LB）跨语言语义搜索的多语言嵌入。我们收集了不同时期的历史卢森堡语新闻文章，并使用GPT-4o进行句子切分和翻译，为每个语言对生成20,000条平行训练句。此外，我们构建了一个语义搜索（Historical LB Bitext Mining，历史卢森堡语双语文本挖掘）评估集，发现现有模型在历史卢森堡语的跨语言搜索上表现不佳。利用我们的历史平行训练数据及额外的现代平行训练数据，我们通过对比学习或知识蒸馏适配了若干多语言嵌入模型，使所有模型的准确率显著提升。我们发布了适配后的模型以及历史卢森堡语-德语/法语/英语双语文本，以支持进一步研究。
英文摘要：
The growing volume of digitized historical texts requires effective semantic search using text embeddings. However, pre-trained multilingual models face challenges with historical content due to OCR noise and outdated spellings. This study examines multilingual embeddings for cross-lingual semantic search in historical Luxembourgish (LB), a low-resource language. We collect historical Luxembourgish news articles from various periods and use GPT-4o for sentence segmentation and translation, generating 20,000 parallel training sentences per language pair. Additionally, we create a semantic search (Historical LB Bitext Mining) evaluation set and find that existing models perform poorly on cross-lingual search for historical Luxembourgish. Using our historical and additional modern parallel training data, we adapt several multilingual embedding models through contrastive learning or knowledge distillation and increase accuracy significantly for all models. We release our adapted models and historical Luxembourgish-German/French/English bitexts to support further research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
