arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.11289cs.CLcs.AI
收藏

LP Data Pipeline:面向大型语言模型的轻量级、目的驱动数据管道

LP Data Pipeline: Lightweight, Purpose-driven Data Pipeline for Large Language Models

Upstage AI(上 stage 人工智能)

Yungi Kim, Hyunsoo Ha, Seonghoon Yang, Sukyung Lee, Jihoo Kim, Chanjun Park

AI总结:

针对LLM数据集构建过度依赖GPU导致成本高、可及性低的问题,提出完全基于CPU运行的轻量级、目的驱动数据管道(LP Data Pipeline),在保障高质量的同时降低时间与成本,并支持特定领域和语言的数据定制。

摘要:

为大型语言模型(LLMs)创建高质量、大规模的数据集通常依赖于资源密集型、GPU加速的模型进行质量过滤,使得该过程耗时且昂贵。这种对GPU的依赖限制缺乏大量计算基础设施的机构的可及性。为解决此问题,我们引入了轻量级、目的驱动的(LP)Data Pipeline,这是一个完全在CPU上运行以简化数据集提取、过滤和整理过程的框架。基于我们的四项核心原则,LP Data Pipeline在保持高数据质量的同时,显著减少了准备时间和成本。重要的是,我们的管道能够创建针对特定领域和语言定制的目的驱动数据集,从而增强LLMs在专业环境中的适用性。我们预计我们的管道将降低LLM开发的门槛,使广泛的机构能更轻松地使用LLMs。

英文摘要:

Creating high-quality, large-scale datasets for large language models (LLMs) often relies on resource-intensive, GPU-accelerated models for quality filtering, making the process time-consuming and costly. This dependence on GPUs limits accessibility for organizations lacking significant computational infrastructure. To address this issue, we introduce the Lightweight, Purpose-driven (LP) Data Pipeline, a framework that operates entirely on CPUs to streamline the processes of dataset extraction, filtering, and curation. Based on our four core principles, the LP Data Pipeline significantly reduces preparation time and cost while maintaining high data quality. Importantly, our pipeline enables the creation of purpose-driven datasets tailored to specific domains and languages, enhancing the applicability of LLMs in specialized contexts. We anticipate that our pipeline will lower the barriers to LLM development, enabling a wide range of organizations to access LLMs more easily.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑