CDS：基于认知诊断理论的知识组件驱动数据合成
CDS: Knowledge Component-Driven Data Synthesis Guided by Cognitive Diagnosis Theory
AI总结：
提出CDS方法，利用认知诊断理论在知识组件层面刻画LLM画像，指导弱点定向数据合成，在代码、数学和学术基准上分别提升6.00%、13.10%和5.43%。
摘要：
大型语言模型（LLMs）已取得显著进展，但任务复杂性的增加和更高的性能要求凸显了持续改进的必要性。一些方法利用基于评估结果由先进LLMs生成的合成数据来训练模型。然而，传统评估方法无法提供LLMs的详细、细粒度画像，限制了其对数据合成的指导。本文提出认知诊断合成（CDS）方法，该方法引入受认知诊断理论（CDT）启发的诊断过程，以细化评估结果并在知识组件层面刻画模型画像。基于这些诊断，我们提出了两种针对弱点定向数据合成的诊断-合成策略。此外，我们提出了一种增强的数据增强和选择流程，以提高合成数据的质量和多样性。我们在多个开源模型上的实验显示，在多个基准上取得了显著改进，在代码生成上提升高达6.00%，数学推理上提升13.10%，学术考试上提升5.43%。代码和数据可在GitHub上获取。
英文摘要：
Large Language Models (LLMs) have achieved significant advancements, but the increasing complexity of tasks and higher performance demands highlight the need for continuous improvement. Some approaches utilize synthetic data generated by advanced LLMs based on evaluation results to train models. However, conventional evaluation methods fail to provide detailed, fine-grained profiles of LLMs, limiting their guidance for data synthesis. In this paper, we introduce the Cognitive Diagnostic Synthesis (CDS) method, which incorporates a diagnostic process inspired by Cognitive Diagnosis Theory (CDT) to refine evaluation results and characterize model profiles at the knowledge component level. Based on these diagnostics, we propose two diagnosis-synthesis strategies for weakness-targeted data synthesis. Additionally, we present an enhanced data augmentation and selection pipeline to improve the quality and diversity of synthesized data. Our experiments with several open-source models show significant improvements across multiple benchmarks, achieving up to 6.00% improvement in code generation, 13.10% in mathematical reasoning, and 5.43% in academic exams. Code and data are available on GitHub.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
