Extending the Vocabulary of Fictional Languages using Neural Networks

统计规律与语言学揭示自然对话与虚构对话中的意义

Statistical laws and linguistics differ in naturalistic video and fictional conversations

研究通过统计规律分析自然和虚构对话中词汇规模与词性关系，揭示语言特征对对话动态的影响。

Extending the Subwording Model of Multilingual Pretrained Models for New Languages

Multilingual pretrained models are effective for machine translation and cross-lingual processing because they contain multiple languages in one model. However, they are pretrained after their tokenizers are fixed; therefore it is difficult to change the vocabulary after pretraining. When we extend the pretrained models to new languages, we must modify the tokenizers simultaneously. In this paper, we add new subwords to the SentencePiece tokenizer to apply a multilingual pretrained model to new languages (Inuktitut in this paper). In our experiments, we segmented Inuktitut sentences into subwo…

利用来自可比用户生成内容的词嵌入构建跨语言消费者健康词汇表

Constructing Cross-lingual Consumer Health Vocabulary with Word-Embedding from Comparable User Generated Content

本文提出一种基于可比用户生成内容和词嵌入对齐的跨语言自动术语识别框架，仅需原始语料和少量医学翻译即可将英语消费者健康词汇表扩展至中文，并验证其优于大型语言模型。

面向跨语言LLM适应的持续预训练：增强日语能力

Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities

本研究通过扩展Llama 2词汇表并进行大规模日语网络语料库持续预训练，构建了增强日语能力的LLM——Swallow，其性能优于从头训练的模型，并验证了词汇扩展与平行语料库的有效性。

IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initialization

We present IndoBERTweet, the first large-scale pretrained model for Indonesian Twitter that is trained by extending a monolingually-trained Indonesian BERT model with additive domain-specific vocabulary. We focus in particular on efficient model adaptation under vocabulary mismatch, and benchmark different ways of initializing the BERT embedding layer for new word types. We find that initializing with the average BERT subword embedding makes pretraining five times faster, and is more effective than proposed methods for vocabulary adaptation in terms of extrinsic evaluation over seven Twitter-b…

语言不平衡在跨语言泛化中的作用：来自克隆语言实验的见解

The Role of Language Imbalance in Cross-lingual Generalisation: Insights from Cloned Language Experiments

本研究通过克隆语言实验发现，训练中的语言不平衡（如90/10分割）能提升低频语言性能并增强跨语言对齐，且随规模扩大效果增强，据此设计了无需改变数据即可提升多语言性能的训练方案。