arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.08800cs.CL
收藏

OpenGPT-X模型家族的数据处理

Data Processing for the OpenGPT-X Model Family

Fraunhofer IAIS(弗劳恩霍夫人工智能研究所);Fraunhofer IIS(弗劳恩霍夫信息处理研究所);DFKI(德意志国防科研机构)
Nicolo' Brandizzi, Hammam Abdelwahab, Anirban Bhowmick, Lennard Helmer, Benny Jörg Stein, Pavel Denisov, Qasid Saleem, Michael Fromm, Mehdi Ali, Richard Rutmann… 展开作者

Nicolo' Brandizzi, Hammam Abdelwahab, Anirban Bhowmick, Lennard Helmer, Benny Jörg Stein, Pavel Denisov, Qasid Saleem, Michael Fromm, Mehdi Ali, Richard Rutmann, Farzad Naderi, Mohamad Saif Agy, Alexander Schwirjow, Fabian Küch, Luzian Hahn, Malte Ostendorff, Pedro Ortiz Suarez, Georg Rehm, Dennis Wegener, Nicolas Flores-Herr, Joachim Köhler, Johannes Leveling

AI总结:

本文介绍了OpenGPT-X项目的大规模多语言大语言模型数据准备流水线,区分精选与网络数据并开发专门算法,同时深入分析数据集以符合欧洲法规,为未来工作提供经验。

摘要:

本文全面介绍了为OpenGPT-X项目开发的数据准备流水线,该项目是一项旨在创建开放且高性能的多语言大语言模型(LLM)的大规模计划。该项目的目标是提供覆盖所有主要欧洲语言的模型,特别关注在欧盟内的实际应用。我们解释了所有的数据处理步骤,从数据选择和需求定义开始,直到准备最终过滤后的数据。我们区分了精选数据和网络数据,因为这两个类别由不同的流水线处理,精选数据经过最少的过滤,而网络数据需要大量的过滤和去重。这种区分指导了为两个流水线开发专门的算法解决方案。除了描述处理方法外,我们还对数据集进行了深入分析,提高了透明度并符合欧洲数据法规。最后,我们分享了项目期间面临的关键见解和挑战,为未来LLM大规模多语言数据准备工作提供了建议。

英文摘要:

This paper presents a comprehensive overview of the data preparation pipeline developed for the OpenGPT-X project, a large-scale initiative aimed at creating open and high-performance multilingual large language models (LLMs). The project goal is to deliver models that cover all major European languages, with a particular focus on real-world applications within the European Union. We explain all data processing steps, starting with the data selection and requirement definition to the preparation of the final filtered data. We distinguish between curated data and web data, as each of these categories is handled by distinct pipelines, with curated data undergoing minimal filtering and web data requiring extensive filtering and deduplication. This distinction guided the development of specialized algorithmic solutions for both pipelines. In addition to describing the processing methodologies, we provide an in-depth analysis of the datasets, increasing transparency and alignment with European data regulations. Finally, we share key insights and challenges faced during the project, offering recommendations for future endeavors in large-scale multilingual data preparation for LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑