CroissantLLM: A Truly Bilingual French-English Language Model
Illuin Technology(伊卢因科技);Unbabel(昂巴贝尔公司);Equall(伊阔尔公司);Diabolocom(迪亚博洛科姆公司);MICS, CentraleSupélec(中央理工学院MICS实验室);Instituto de Telecomunicações, Lisboa(里斯本电信研究所);INESC-ID, Lisboa(里斯本INESC-ID研究所);Instituto Superior Técnico, Universidade de Lisboa(里斯本大学高等技术学院);Sorbonne Université, CNRS, ISIR, Paris(巴黎索邦大学法国国家科学研究中心智能与机器人系统研究所);Imperial College London(伦敦帝国学院);Language Technologies Institute, Carnegie Mellon University(卡内基梅隆大学语言技术研究所)
Manuel Faysse, Patrick Fernandes, Nuno M. Guerreiro, António Loison, Duarte M. Alves, Caio Corro, Nicolas Boizard, João Alves, Ricardo Rei, Pedro H. Martins, An… 展开作者
Manuel Faysse, Patrick Fernandes, Nuno M. Guerreiro, António Loison, Duarte M. Alves, Caio Corro, Nicolas Boizard, João Alves, Ricardo Rei, Pedro H. Martins, Antoni Bigata Casademunt, François Yvon, André F. T. Martins, Gautier Viaud, Céline Hudelot, Pierre Colombo
We introduce CroissantLLM, a 1.3B language model pretrained on a set of 3T English and French tokens, to bring to the research and industrial community a high-performance, fully open-sourced bilingual model that runs swiftly on consumer-grade local hardware. To that end, we pioneer the approach of training an intrinsically bilingual model with a 1:1 English-to-French pretraining data ratio, a custom tokenizer, and bilingual finetuning datasets. We release the training dataset, notably containing a French split with manually curated, high-quality, and varied data sources. To assess performance outside of English, we craft a novel benchmark, FrenchBench, consisting of an array of classification and generation tasks, covering various orthogonal aspects of model performance in the French Language. Additionally, rooted in transparency and to foster further Large Language Model research, we release codebases, and dozens of checkpoints across various model sizes, training data distributions, and training steps, as well as fine-tuned Chat models, and strong translation models. We evaluate our model through the FMTI framework, and validate 81 % of the transparency criteria, far beyond the scores of even most open initiatives. This work enriches the NLP landscape, breaking away from previous English-centric work in order to strengthen our understanding of multilinguality in language models.
Multilingual LLMs Struggle to Link Orthography and Semantics in Bilingual Word Processing
Department of Electrical Engineering, Indian Institute of Technology Delhi(印度理工学院德里电气工程系);Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi(印度理工学院德里人工智能学院)
From Traditional Taggers to LLMs: A Comparative Study of POS Tagging for Medieval Romance Languages
Bavarian Academy of Sciences (BAdW)(巴伐利亚科学院);Department of Statistics, LMU Munich, Germany(慕尼黑大学统计系);Munich Center for Machine Learning (MCML), LMU Munich, Germany(慕尼黑机器学习中心)
MultiMed-ST: Large-scale Many-to-many Multilingual Medical Speech Translation
University of Toronto(多伦多大学);University Health Network(大学健康网络);Knovel Engineering Lab(Knovel 工程实验室);Hanoi University of Science and Technology(河内科技大学);KU Leuven(鲁汶大学);Eötvös Loránd University(罗兰大学);HCMC Open University(胡志明市开放大学);IÉSEG School of Management(IÉSEG 管理学院);Technische Universität Dortmund(多特蒙德工业大学);University of Hertfordshire(赫特福德大学);UC Berkeley(加州大学伯克利分校);New Jersey Institute of Technology(新泽西理工学院)
How Does Language Influence Documentation Workflow? Unsupervised Word Discovery Using Translations in Multiple Languages
Laboratoire d’Informatique de Grenoble (LIG)(格勒诺布尔计算机实验室);UGA(格勒诺布尔阿尔卑斯大学);G-INP(格勒诺布尔国立理工学院);CNRS(法国国家科学研究中心);INRIA(法国国家信息与自动化研究所);University of Sheffield(谢菲尔德大学);UFRGS(南里奥格兰德联邦大学)
对于语言文档举措,转录是一种昂贵的资源:一分钟的音频估计需要语言学家平均一个半小时的工作(Austin和Sallabank,2013年)。最近,在资源丰富的语言中收集对齐的翻译成为确保记录的事后可解释性的流行解决方案(Adda et al. 2016)。在本文中,我们调查语言相关的影响,在自动方法计算语言文档。我们将双语Mboshi-French平行语料库(Godard et al. 2017)翻译成其他四种语言,并执行双语根无监督单词发现。我们的研究结果暗示了资源丰富的语言对输出质量的影响。然而,通过结合不同双语模型学习到的信息,我们只能略微提高分割的质量。
Pirá: A Bilingual Portuguese-English Dataset for Question-Answering about the Ocean
Escola de Artes, Ciências e Humanidades Universidade de São Paulo(圣保罗大学艺术、科学与人文学院);Instituto de Estudos Avançados Universidade de São Paulo(圣保罗大学高等研究院);Escola Politécnica Universidade de São Paulo(圣保罗大学理工学院)
口语翻译(spoken language translation,缩写为EMT)中的最近的工作已经尝试在学习或解码期间不使用源语言转录的情况下构建端到端的语音到文本的翻译。然而,虽然大量的并行文本(例如Europarl、OpenSubtitles)可用于训练机器翻译系统,但是不存在包括与目标语言的文本对齐的源语言的语音的大型(100 h)和开源并行语料库。本文试图填补这一空白,增加现有的(单语)语料库:LibriSpeech。该语料库用于自动语音识别,来自LibriVox项目的有声读物,并经过仔细的分段和对齐。在从LibriSpeech中收集与英语有声读物相对应的法语电子书后,我们将句子级别的语音片段与各自的翻译对齐,并获得236 h的可用并行数据。本文介绍了处理的细节,以及进行人工评估的一个小的子集的语料库。该评估表明,自动对齐分数与双语对齐质量的人类判断合理相关。我们相信,这个语料库(这是在线提供)是有用的直接语音翻译或更一般的口语翻译实验的可复制实验。
Examining Gender Bias in Languages with Grammatical Gender
University of California Los Angeles(加州大学洛杉矶分校);University of Southern California(南加州大学);University of Pennsylvania(宾夕法尼亚大学);Johns Hopkins University(约翰斯·霍普金斯大学)