DACIP-RC：通过商业对话阅读理解实现领域自适应持续指令预训练
DACIP-RC: Domain Adaptive Continual Instruction Pre-Training via Reading Comprehension on Business Conversations
AI总结：
针对小型语言模型在商业对话任务中零样本能力不足及微调导致灾难性遗忘的问题，提出DACIP-RC持续预训练方法，通过阅读理解生成指令，显著提升会议摘要等任务的泛化性能。
摘要：
大型语言模型（LLMs）的快速发展使其能够在真实工业场景中应用于各种自然语言处理任务。然而，大规模LLMs的高推理成本使其部署不切实际，因此需要使用较小的模型。尽管小型LLMs效率较高，但它们在跨不同领域的零样本指令跟随能力上有所欠缺，限制了其对动态用户需求的适应性。传统的微调方法会引发灾难性遗忘，从而加剧这一问题，降低模型对未见任务的泛化能力。在本文中，我们提出了通过阅读理解实现领域自适应持续指令预训练（DACIP-RC），这是一种持续预训练技术，旨在增强小型LLMs在商业对话任务中的领域适应性。与依赖下一个词元预测的传统预训练方法不同，DACIP-RC通过对对话记录进行阅读理解来生成多样化的任务指令和响应，从而实现更好的指令泛化。我们的实证评估表明，DACIP-RC显著提升了在广泛商业对话任务（包括会议摘要、行动项生成和通话目的识别）中的零样本泛化能力。据我们所知，这是首个将指令预训练应用于商业对话数据的工作，为行业如何利用专有数据集进行领域自适应提供了见解。
英文摘要：
The rapid advancements in Large Language Models (LLMs) have enabled their adoption in real-world industrial scenarios for various natural language processing tasks. However, the high inference cost of large-scale LLMs makes their deployment impractical, necessitating the use of smaller models. Despite their efficiency, smaller LLMs lack robust zero-shot instruction-following capabilities across diverse domains, limiting their adaptability to dynamic user requirements. Traditional fine-tuning approaches exacerbate this issue by inducing catastrophic forgetting, reducing the model's generalization ability for unseen tasks. In this paper, we propose Domain Adaptive Continual Instruction Pre-Training via Reading Comprehension (DACIP-RC), a continual pre-training technique that enhances smaller LLMs' domain adaptability for business conversational tasks. Unlike conventional pre-training approaches that rely on next-token prediction, DACIP-RC generates diverse task instructions and responses via reading comprehension on conversation transcripts, enabling better instruction generalization. Our empirical evaluations demonstrate that DACIP-RC significantly improves zero-shot generalization across a wide range of business conversational tasks, including meeting summarization, action item generation, and call purpose identification. To the best of our knowledge, this is the first work to apply instruction pre-training on business conversational data, providing insights into how industries can leverage proprietary datasets for domain adaptation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
