重新思考数据：迈向性能更优的领域特定小语言模型
Rethinking Data: Towards Better Performing Domain-Specific Small Language Models
AI总结：
本文提出通过在LM训练各阶段提升数据质量（含文本块提取、轻量级重排序器CRR及模型合并）来微调小型语言模型，使其在多项选择题回答任务中实现高准确率且具成本效益。
摘要：
在领域特定数据上对大型语言模型（LLMs）进行下游任务微调已展现出巨大潜力。然而，此类LLMs的商业用途受限于其大规模部署所需的高昂计算成本。另一方面，小型语言模型成本效益更高，但在类似设置下性能欠佳。本文提出了微调小型LM的方法，使其在多项选择题回答任务中达到高准确率。我们通过在LM训练流程的每个阶段提升数据质量来实现这一点。具体而言，我们首先进行数据结构化，提取紧凑且语义明确的文本块供检索器使用，使LM能更高效地消化知识。其次，通过训练轻量级块重排序器生成更准确的相对相关性块分数来改进检索上下文。最后，通过合并在不同数据子集上以不同参数微调的模型来提升模型泛化能力。我们提供了详细流程描述及相应实验结果，证明各项技术的改进效果。
英文摘要：
Fine-tuning of Large Language Models (LLMs) for downstream tasks, performed on domain-specific data has shown significant promise. However, commercial use of such LLMs is limited by the high computational cost required for their deployment at scale. On the other hand, small Language Models (LMs) are much more cost effective but have subpar performance in a similar setup. This paper presents our approach to finetuning a small LM, that reaches high accuracy in multiple choice question answering task. We achieve this by improving data quality at each stage of the LM training pipeline. In particular, we start with data structuring resulting in extraction of compact, semantically meaningful text chunks used by a retriever. This allows more efficient knowledge digestion by the LM. Further, we improve the retrieved context by training a lightweight Chunk Re-Ranker (CRR) that generates more accurate relative relevance chunk scores. Finally, we improve the model generalization ability by merging the models fine-tuned with different parameters on different data subsets. We present detailed procedure descriptions, and corresponding experimental findings that show the improvements of each one of the proposed techniques.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
