利用闭源多语言嵌入实现低资源语言的自动句子对齐
Leveraging Closed-Access Multilingual Embedding for Automatic Sentence Alignment in Low Resource Languages
AI总结：
本研究提出一种利用闭源 Cohere 多语言嵌入的平行句子对齐器，有效解决低资源语言数据获取难题，在 FLORES 和 MAFAND-MT 上 F1 分数远超 LASER，并显著提升翻译模型 BLEU 分数。
摘要：
高质量平行数据在机器翻译中的重要性早已确定，但对于世界上大多数语言，始终很难获得足够数量的此类数据，主要是因为相关成本以及这些语言的可及性不足。尽管可以通过自动方法从在线文章中获取平行数据集，但取证调查发现了很多与质量相关的问题，例如错位和错误的语言代码。在本研究中，我们提出了一种简单但高质量的平行句子对齐器，它谨慎地利用了闭源访问的 Cohere 多语言嵌入，该解决方案在刚刚结束的 #CoHereAIHack 2023 挑战赛中排名第二（见 https://ai6lagos.devpost.com）。所提出的方法在 FLORES 和 MAFAND-MT 上分别取得了 94.96 和 54.83 的 F1 分数，而 LASER 则分别为 3.64 和 0.64。当使用生成的数据集与 MAFAND-MT 数据集一起训练翻译模型时，我们的方法还比 LASER 提高了 5 个以上的 BLEU 分数。我们的代码和数据可在此处用于研究目的（https://github.com/abumafrim/Cohere-Align）。
英文摘要：
The importance of qualitative parallel data in machine translation has long been determined but it has always been very difficult to obtain such in sufficient quantity for the majority of world languages, mainly because of the associated cost and also the lack of accessibility to these languages. Despite the potential for obtaining parallel datasets from online articles using automatic approaches, forensic investigations have found a lot of quality-related issues such as misalignment, and wrong language codes. In this work, we present a simple but qualitative parallel sentence aligner that carefully leveraged the closed-access Cohere multilingual embedding, a solution that ranked second in the just concluded #CoHereAIHack 2023 Challenge (see https://ai6lagos.devpost.com). The proposed approach achieved $94.96$ and $54.83$ f1 scores on FLORES and MAFAND-MT, compared to $3.64$ and $0.64$ of LASER respectively. Our method also achieved an improvement of more than 5 BLEU scores over LASER, when the resulting datasets were used with MAFAND-MT dataset to train translation models. Our code and data are available for research purposes here (https://github.com/abumafrim/Cohere-Align).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
