EcomGPT-CT：基于半结构化数据的电子商务大语言模型持续预训练
EcomGPT-CT: Continual Pre-training of E-commerce Large Language Models with Semi-structured Data
AI总结：
针对大语言模型在电商领域缺乏领域知识等问题，提出利用无标注通用与电商语料进行持续预训练，并设计数据混合策略，实验验证了其有效性与少样本及零样本性能提升。
摘要：
在大规模语料上预训练的大语言模型（LLMs）在各种自然语言处理任务中表现出色。然而，将这些模型应用于特定领域仍面临重大挑战，例如缺乏领域知识、利用领域知识的能力有限以及对特定领域数据格式的适应不足。考虑到从头训练大语言模型的昂贵成本以及特定领域内标注数据的稀缺性，本工作以电子商务领域为例，聚焦于大语言模型的领域特定持续预训练。具体而言，我们探讨了使用无标注的通用语料和电子商务语料对大语言模型进行持续预训练的影响。此外，我们设计了一种不同数据源之间的混合策略，以更好地利用电子商务半结构化数据。我们构建了多个任务来评估大语言模型在电子商务领域的少样本上下文学习能力以及指令微调后的零样本性能。实验结果表明，电子商务大语言模型的持续预训练是有效的，并且我们设计的数据混合策略也是有效的。
英文摘要：
Large Language Models (LLMs) pre-trained on massive corpora have exhibited remarkable performance on various NLP tasks. However, applying these models to specific domains still poses significant challenges, such as lack of domain knowledge, limited capacity to leverage domain knowledge and inadequate adaptation to domain-specific data formats. Considering the exorbitant cost of training LLMs from scratch and the scarcity of annotated data within particular domains, in this work, we focus on domain-specific continual pre-training of LLMs using E-commerce domain as an exemplar. Specifically, we explore the impact of continual pre-training on LLMs employing unlabeled general and E-commercial corpora. Furthermore, we design a mixing strategy among different data sources to better leverage E-commercial semi-structured data. We construct multiple tasks to assess LLMs' few-shot In-context Learning ability and their zero-shot performance after instruction tuning in E-commerce domain. Experimental results demonstrate the effectiveness of continual pre-training of E-commerce LLMs and the efficacy of our devised data mixing strategy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
