我们探讨了元学习在英语、俄语和中文的语言无关无监督分词问题中的可能性。我们实现了元学习方法，用于自动确定早期工作中提出的无监督分词模型的超参数，该方法依赖于多种独立于人类的适应度函数，如归一化反熵、压缩因子和跨分割 F1 分数，以及这三种指标的加性和乘性复合组合，并针对常规 F1 分词分数进行了测试。我们发现，对于英语和俄语，后者与前三项指标的加性组合之间存在相当好的相关性。对于中文，我们发现 F1 分数与压缩因子之间存在显著相关性。我们的结果表明，对低资源语言和已消亡语言进行鲁棒的无监督分词是可能的，并使我们能够从具有不同结构优化方案的有效符号通信码演化的角度来思考人类语言，这些方案在不同的人类文化中演化而来。
英文摘要：
We explore the possibility of meta-learning for the language-independent unsupervised tokenization problem for English, Russian, and Chinese. We implement the meta-learning approach for automatic determination of hyper-parameters of the unsupervised tokenization model proposed in earlier works, relying on various human-independent fitness functions such as normalised anti-entropy, compression factor and cross-split F1 score, as well as additive and multiplicative composite combinations of the three metrics, testing them against the conventional F1 tokenization score. We find a fairly good correlation between the latter and the additive combination of the former three metrics for English and Russian. In case of Chinese, we find a significant correlation between the F 1 score and the compression factor. Our results suggest the possibility of robust unsupervised tokenization of low-resource and dead languages and allow us to think about human languages in terms of the evolution of efficient symbolic communication codes with different structural optimisation schemes that have evolved in different human cultures.
Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of LLMs for Low-Resource NLP
Ghent University（根特大学）；imec；KU Leuven（鲁汶大学）；BeCode asbl；Internet and Data Science Lab (IDLab)（互联网与数据科学实验室（IDLab））；AI & Data Bootcamp Ghent 2024（2024年根特人工智能与数据训练营）
Conditional Unigram Tokenization with Parallel Data
Institute of Formal and Applied Linguistics, Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic（形式与应用语言学研究所，数学与物理系，查尔斯大学，布拉格，捷克共和国）
The Chinese University of Hong Kong（香港中文大学）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）
Cross-lingual word embeddings can be applied to several natural language processing applications across multiple languages. Unlike prior works that use word embeddings based on the Euclidean space, this short paper presents a simple and effective cross-lingual Word2Vec model that adapts to the Poincaré ball model of hyperbolic space to learn unsupervised cross-lingual word representations from a German-English parallel corpus. It has been shown that hyperbolic embeddings can capture and preserve hierarchical relationships. We evaluate the model on both hypernymy and analogy tasks. The proposed…
University of Maryland（马里兰大学）；University of Pennsylvania（宾夕法尼亚大学）
Cross-lingual Hypernymy Detection involves determining if a word in one language ("fruit") is a hypernym of a word in another language ("pomme" i.e. apple in French). The ability to detect hypernymy cross-lingually can aid in solving cross-lingual versions of tasks such as textual entailment and event coreference. We propose BISPARSE-DEP, a family of unsupervised approaches for cross-lingual hypernymy detection, which learns sparse, bilingual word embeddings based on dependency contexts. We show that BISPARSE-DEP can significantly improve performance on this task, compared to approaches based …
The zero-shot cross-lingual ability of models pretrained on multilingual and even monolingual corpora has spurred many hypotheses to explain this intriguing empirical result. However, due to the costs of pretraining, most research uses public models whose pretraining methodology, such as the choice of tokenization, corpus size, and computational budget, might differ drastically. When researchers pretrain their own models, they often do so under a constrained budget, and the resulting models might underperform significantly compared to SOTA models. These experimental differences led to various …
One For All & All For One: Bypassing Hyperparameter Tuning with Model Averaging For Cross-Lingual Transfer
Center For Artificial Intelligence and Data Science, University of Würzburg（维尔茨堡大学人工智能与数据科学中心）；Language Technology Lab, University of Cambridge（剑桥大学语言技术实验室）
National University of Ireland（爱尔兰国立大学）；Aylien Ltd.（Aylien有限公司）；University of Cambridge（剑桥大学）；University of Copenhagen（哥本哈根大学）
Cross-lingual representations of words enable us to reason about word meaning in multilingual contexts and are a key facilitator of cross-lingual transfer when developing natural language processing models for low-resource languages. In this survey, we provide a comprehensive typology of cross-lingual word embedding models. We compare their data requirements and objective functions. The recurring theme of the survey is that many of the models presented in the literature optimize for the same objectives, and that seemingly different models are often equivalent modulo optimization strategies, hy…
Ominor AI；University of the Witwatersrand（威特沃特斯兰德大学）
Stemming from the limited availability of datasets and textual resources for low-resource languages such as isiZulu, there is a significant need to be able to harness knowledge from pre-trained models to improve low resource machine translation. Moreover, a lack of techniques to handle the complexities of morphologically rich languages has compounded the unequal development of translation models, with many widely spoken African languages being left behind. This study explores the potential benefits of transfer learning in an English-isiZulu translation framework. The results indicate the value…
In this paper, we reproduce the experiments of Artetxe et al. (2018b) regarding the robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. We show that the reproduction of their method is indeed feasible with some minor assumptions. We further investigate the robustness of their model by introducing four new languages that are less similar to English than the ones proposed by the original paper. In order to assess the stability of their model, we also conduct a grid search over sensible hyperparameters. We then propose key recommendations applicable to an…
Universidade de Vigo（维戈大学）；Cardiff University（卡迪夫大学）
Word embeddings have become a standard resource in the toolset of any Natural Language Processing practitioner. While monolingual word embeddings encode information about words in the context of a particular language, cross-lingual embeddings define a multilingual space where word embeddings from two or more languages are integrated together. Current state-of-the-art approaches learn these embeddings by aligning two disjoint monolingual vector spaces through an orthogonal transformation which preserves the structure of the monolingual counterparts. In this work, we propose to apply an addition…
The growth of cross-lingual pre-trained models has enabled NLP tools to rapidly generalize to new languages. While these models have been applied to tasks involving entities, their ability to explicitly predict typological features of these entities across languages has not been established. In this paper, we present a unified cross-lingual fine-grained entity typing model capable of handling over 100 languages and analyze this model's ability to generalize to languages and entities unseen during training. We train this model on cross-lingual training data collected from Wikipedia hyperlinks i…
Transformers that are pre-trained on multilingual corpora, such as, mBERT and XLM-RoBERTa, have achieved impressive cross-lingual transfer capabilities. In the zero-shot transfer setting, only English training data is used, and the fine-tuned model is evaluated on another target language. While this works surprisingly well, substantial variance has been observed in target language performance between different fine-tuning runs, and in the zero-shot setup, no target-language development data is available to select among multiple fine-tuned models. Prior work has relied on English dev data to se…
Silo AI；University of Lille（里尔大学）；Institute of Mathematics and Applications（数学与应用研究所）；KIIT University（KIIT大学）；Ravenshaw University（雷文肖大学）
This paper presents the first publicly available treebank of Odia, a morphologically rich low resource Indian language. The treebank contains approx. 1082 tokens (100 sentences) in Odia selected from "Samantar", the largest available parallel corpora collection for Indic languages. All the selected sentences are manually annotated following the ``Universal Dependency (UD)" guidelines. The morphological analysis of the Odia treebank was performed using machine learning techniques. The Odia annotated treebank will enrich the Odia language resource and will help in building language technology to…
School of Information Science and Technology, Vidyasirimedhi Institution of Science and Technology（威集玛迪科学技术大学信息科学技术学院）；PyThaiNLP
Transformer-based language models, more specifically BERT-based architectures have achieved state-of-the-art performance in many downstream tasks. However, for a relatively low-resource language such as Thai, the choices of models are limited to training a BERT-based model based on a much smaller dataset or finetuning multi-lingual models, both of which yield suboptimal downstream performance. Moreover, large-scale multi-lingual pretraining does not take into account language-specific features for Thai. To overcome these limitations, we pretrain a language model based on RoBERTa-base architect…
Johns Hopkins University（约翰斯·霍普金斯大学）；University of Chinese Academy of Sciences（中国科学院大学）；Amazon（亚马逊）；International Institute of Information Technology（国际信息技术学院）；Stanford University（斯坦福大学）；Brno University of Technology（布尔诺理工大学）
Acoustic unit discovery (AUD) is a process of automatically identifying a categorical acoustic unit inventory from speech and producing corresponding acoustic unit tokenizations. AUD provides an important avenue for unsupervised acoustic model training in a zero resource setting where expert-provided linguistic knowledge and transcribed speech are unavailable. Therefore, to further facilitate zero-resource AUD process, in this paper, we demonstrate acoustic feature representations can be significantly improved by (i) performing linear discriminant analysis (LDA) in an unsupervised self-trained…
Free Lunch: Robust Cross-Lingual Transfer via Model Checkpoint Averaging
Center For Artificial Intelligence and Data Science, University of Würzburg（维尔茨堡大学人工智能与数据科学中心）；Language Technology Lab, University of Cambridge（剑桥大学语言技术实验室）
Beyond Fertility: Analyzing STRR as a Metric for Multilingual Tokenization Evaluation
University of Alberta（阿尔伯塔大学）；Princess Nourah Bint Abdulrahman University（努拉·阿卜杜勒拉赫曼王后大学）；Dialpad Qatar Computing Research Institute（阿联酋迪拜计算研究所）；Amazon AGI（亚马逊人工智能研究所）
Modern language models mostly take sub-words as input, a design that balances the trade-off between vocabulary size, number of parameters, and performance. However, sub-word tokenization still has disadvantages like not being robust to noise and difficult to generalize to new languages. Also, the current trend of scaling up models reveals that larger models require larger embeddings but that makes parallelization hard. Previous work on image classification proves splitting raw input into a sequence of chucks is a strong, model-agnostic inductive bias. Based on this observation, we rethink the …
University of Zagreb（萨格勒布大学）；Faculty of Humanities and Social Sciences（人文与社会科学学院）
This article presents the strategy for developing a platform containing Language Processing Chains for European Union languages, consisting of Tokenization to Parsing, also including Named Entity recognition andwith addition ofSentiment Analysis. These chains are part of the first step of an event-centric knowledge processing pipeline whose aim is to process multilingual media information about major events that can cause an impactin Europe and the rest of the world. Due to the differences in terms of availability of language resources for each language, we have built this strategy in three st…
On the Analysis of Cross-Lingual Prompt Tuning for Decoder-based Multilingual Model
Seoul National University（首尔大学）；NAVER Cloud（NAVER云）；NAVER AI Lab（NAVER人工智能实验室）；University of Richmond（里士满大学）；Artificial Intelligence Institute, Seoul National University（首尔大学人工智能研究所）
Large pretrained language models (LMs) have become the central building block of many NLP applications. Training these models requires ever more computational resources and most of the existing models are trained on English text only. It is exceedingly expensive to train these models in other languages. To alleviate this problem, we introduce a novel method -- called WECHSEL -- to efficiently and effectively transfer pretrained LMs to new languages. WECHSEL can be applied to any model which uses subword-based tokenization and learns an embedding for each subword. The tokenizer of the source mo…
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；School of Computer Science and Engineering, Central South University（中南大学计算机科学与工程学院）；College of Information Science and Technology and Artificial Intelligence, Nanjing Forestry University（南京林业大学信息科学技术与人工智能学院）
IXA NLP Group（IXA自然语言处理研究组）；University of the Basque Country (UPV/EHU)（巴斯克大学）
While modern machine translation has relied on large parallel corpora, a recent line of work has managed to train Neural Machine Translation (NMT) systems from monolingual corpora only (Artetxe et al., 2018c; Lample et al., 2018). Despite the potential of this approach for low-resource settings, existing systems are far behind their supervised counterparts, limiting their practical interest. In this paper, we propose an alternative approach based on phrase-based Statistical Machine Translation (SMT) that significantly closes the gap with supervised systems. Our method profits from the modular …
MoCE: Adaptive Mixture of Contextualization Experts for Byte-based Neural Machine Translation
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Key Laboratory of AI Safety, Chinese Academy of Sciences（中国科学院人工智能安全重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Washington University in St. Louis（圣路易斯华盛顿大学）
