arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.05314cs.ITcs.AIcs.LGmath.IT
收藏

Tele-LLMs:面向电信领域的一系列专用大语言模型

Tele-LLMs: A Series of Specialized Large Language Models for Telecommunications

Yale University(耶鲁大学);Amazon Web Services(亚马逊网络服务)

Ali Maatouk, Kenny Chirino Ampudia, Rex Ying, Leandros Tassiulas

AI总结:

本文针对电信领域缺乏专用大语言模型的问题,构建了Tele-Data和Tele-Eval数据集,系统研究适配训练技术,并开源了参数量1B到8B的Tele-LLMs系列模型,在电信任务上超越通用模型且避免灾难性遗忘。

摘要:

大语言模型(LLMs)的出现对从自然语言处理到医学、金融等多个领域产生了深远影响。然而,尽管其迅速普及,LLMs在电信领域的应用仍然有限,通常依赖缺乏领域专用性的通用模型。这种专用性的缺乏导致性能不佳,尤其是在处理电信特有的技术术语及其相关数学表示时。本文通过首先创建并发布Tele-Data(一个从相关来源整理的电信材料综合数据集)和Tele-Eval(一个面向该领域的大规模问答数据集)来弥补这一空白。通过大量实验,我们探索了将LLMs适配到电信领域的最有效训练技术,包括考察电信各子领域专业知识的划分以及采用参数高效技术。我们还研究了不同规模的模型在适配过程中的表现,并分析了其训练数据对这种表现的影响。利用这些发现,我们开发并开源了Tele-LLMs,这是首个参数量从1B到8B不等、专门面向电信领域的语言模型系列。我们的评估表明,这些模型在Tele-Eval和电信相关文献任务上优于其通用对应模型,同时保留了此前获得的能力,从而避免了灾难性遗忘现象。

英文摘要:

The emergence of large language models (LLMs) has significantly impacted various fields, from natural language processing to sectors like medicine and finance. However, despite their rapid proliferation, the applications of LLMs in telecommunications remain limited, often relying on general-purpose models that lack domain-specific specialization. This lack of specialization results in underperformance, particularly when dealing with telecommunications-specific technical terminology and their associated mathematical representations. This paper addresses this gap by first creating and disseminating Tele-Data, a comprehensive dataset of telecommunications material curated from relevant sources, and Tele-Eval, a large-scale question-and-answer dataset tailored to the domain. Through extensive experiments, we explore the most effective training techniques for adapting LLMs to the telecommunications domain, ranging from examining the division of expertise across various telecommunications aspects to employing parameter-efficient techniques. We also investigate how models of different sizes behave during adaptation and analyze the impact of their training data on this behavior. Leveraging these findings, we develop and open-source Tele-LLMs, the first series of language models ranging from 1B to 8B parameters, specifically tailored for telecommunications. Our evaluations demonstrate that these models outperform their general-purpose counterparts on Tele-Eval and telecommunications-related literature tasks while retaining their previously acquired capabilities, thus avoiding the catastrophic forgetting phenomenon.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑