The need for raw large raw corpora has dramatically increased in recent years with the introduction of transfer learning and semi-supervised learning methods to Natural Language Processing. And while there have been some recent attempts to manually curate the amount of data necessary to train large language models, the main way to obtain this data is still through automatic web crawling. In this paper we take the existing multilingual web corpus OSCAR and its pipeline Ungoliant that extracts and classifies data from Common Crawl at the line level, and propose a set of improvements and automatic annotations in order to produce a new document-oriented version of OSCAR that could prove more suitable to pre-train large generative language models as well as hopefully other applications in Natural Language Processing and Digital Humanities.
Web-crawled data provides a good source of parallel corpora for training machine translation models. It is automatically obtained, but extremely noisy, and recent work shows that neural machine translation systems are more sensitive to noise than traditional statistical machine translation methods. In this paper, we propose a novel approach to filter out noisy sentence pairs from web-crawled corpora via pre-trained language models. We measure sentence parallelism by leveraging the multilingual capability of BERT and use the Generative Pre-training (GPT) language model as a domain filter to bal…
University of Leipzig（莱比锡大学）；University of Paderborn（帕德博恩大学）；University of Wolverhampton（伍尔弗汉普顿大学）
In this paper, we describe the LIDIOMS data set, a multilingual RDF representation of idioms currently containing five languages: English, German, Italian, Portuguese, and Russian. The data set is intended to support natural language processing applications by providing links between idioms across languages. The underlying data was crawled and integrated from various sources. To ensure the quality of the crawled data, all idioms were evaluated by at least two native speakers. Herein, we present the model devised for structuring the data. We also provide the details of linking LIDIOMS to well-k…
mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus
Inria（法国国家信息与自动化技术研究院）；Département d’informatique de l’ENS, CNRS, PSL Research University（巴黎高等师范学院计算机科学系、国家科学研究中心、巴黎综合理工研究学院）；Institut du développement et des ressources en informatique scientifique, CNRS（科学信息发展与资源研究所、国家科学研究中心）；Sorbonne Université（索邦大学）；Common Crawl Foundation（Common Crawl基金会）；Université Paris-Saclay（巴黎萨克雷大学）
Google Research（谷歌研究院）；Masakhane NLP（Masakhane自然语言处理组织）；Turkic Interlingua（突厥语国际语组织）；Haverford College（哈弗福德学院）；RobotsMali（马里机器人组织）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；University of Zambia（赞比亚大学）；Google（谷歌公司）；AIMS-AMMI（非洲数学科学研究所-非洲机器学习硕士项目）；Inria（法国国家信息与自动化研究所）；University of Zurich（苏黎世大学）；Stanford University（斯坦福大学）；Kwame Nkrumah University of Science and Technology（夸梅·恩克鲁玛科技大学）；Sorbonne Université（索邦大学）；Niger-Volta LTI（尼日尔-沃尔特语言技术研究所）；University of Waterloo（滑铁卢大学）；University of Electronic Science and Technology of China（电子科技大学）；University of Notre Dame（圣母大学）；Bayero University Kano（卡诺巴耶罗大学）
With the success of large-scale pre-training and multilingual modeling in Natural Language Processing (NLP), recent years have seen a proliferation of large, web-mined text datasets covering hundreds of languages. We manually audit the quality of 205 language-specific corpora released with five major public datasets (CCAligned, ParaCrawl, WikiMatrix, OSCAR, mC4). Lower-resource corpora have systematic issues: At least 15 corpora have no usable text, and a significant fraction contains less than 50% sentences of acceptable quality. In addition, many are mislabeled or use nonstandard/ambiguous l…
Miðeind ehf.（Miðeind 公司）；University of Iceland（冰岛大学）
We train several language models for Icelandic, including IceBERT, that achieve state-of-the-art performance in a variety of downstream tasks, including part-of-speech tagging, named entity recognition, grammatical error detection and constituency parsing. To train the models we introduce a new corpus of Icelandic text, the Icelandic Common Crawl Corpus (IC3), a collection of high quality texts found online by targeting the Icelandic top-level-domain (TLD). Several other public data sources are also collected for a total of 16GB of Icelandic text. To enhance the evaluation of model performance…
West Greenlandic, known by native speakers as Kalaallisut, is an extremely low-resource polysynthetic language spoken by around 56,000 people in Greenland. Here, we attempt to finetune a pretrained Kalaallisut-to-English neural machine translation (NMT) system using web-crawled pseudoparallel sentences from around 30 multilingual websites. We compile a corpus of over 93,000 Kalaallisut sentences and over 140,000 Danish sentences, then use cross-lingual sentence embeddings and approximate nearest-neighbors search in an attempt to mine near-translations from these corpora. Finally, we translate …
ChineseWebText: Large-scale High-quality Chinese Web Text Extracted with Effective Evaluation Model
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Wuhan AI Research（武汉人工智能研究院）
YODAS v3: Over 1 Million Hours of High-Bandwidth, Stereophonic, Multilingual Speech
Carnegie Mellon University（卡内基梅隆大学）；Keio University（庆应义塾大学）；Tokyo Metropolitan University（东京都立大学）；National Institute of Advanced Industrial Science and Technology (AIST)（产业技术综合研究所）
Adversarial training (AT) is a powerful regularization method for neural networks, aiming to achieve robustness to input perturbations. Yet, the specific effects of the robustness obtained from AT are still unclear in the context of natural language processing. In this paper, we propose and analyze a neural POS tagging model that exploits AT. In our experiments on the Penn Treebank WSJ corpus and the Universal Dependencies (UD) dataset (27 languages), we find that AT not only improves the overall tagging accuracy, but also 1) prevents over-fitting well in low resource languages and 2) boosts t…
Neural machine translation (NMT) has progressed rapidly over the past several years, and modern models are able to achieve relatively high quality using only monolingual text data, an approach dubbed Unsupervised Machine Translation (UNMT). However, these models still struggle in a variety of ways, including aspects of translation that for a human are the easiest - for instance, correctly translating common nouns. This work explores a cheap and abundant resource to combat this problem: bilingual lexica. We test the efficacy of bilingual lexica in a real-world set-up, on 200-language translatio…
