预警即武装：通过诱发失败的探索利用大语言模型进行数据合成
Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Exploration
AI总结：
本文提出 ReverseGen 方法，通过训练提议器自动生成能诱发目标大语言模型失败的查询，并据此构建训练数据以弥补模型缺陷，在安全性、诚实性和数学任务上验证了其数据的高效性与多样性，为任务特定的大语言模型增强提供了新思路。
摘要：
大语言模型（LLMs）因在多样化、高质量的任务特定数据上训练而显著受益，从而在一系列下游应用中取得了令人瞩目的性能。当前方法通常依赖人工标注数据或预定义任务模板来引导强大的大语言模型合成与任务相关的数据，以进行有效的模型训练。然而，这种对手工设计组件的依赖可能会限制生成数据的范围，可能忽略关键的边缘情况或可能挑战模型的新颖场景。在本文中，我们提出了一种新方法 ReverseGen，旨在自动生成能够暴露大语言模型弱点的有效训练样本。具体来说，我们引入了一个专门的提议器（proposer），经过训练以产生导致目标模型生成不令人满意响应的查询。这些诱发失败的查询随后被用于构建训练数据，帮助弥补模型的不足并提升整体性能。我们的方法具有灵活性，可应用于不同规模的模型（3B、7B 和 8B）。我们在三个关键应用（安全性、诚实性和数学）上评估了 ReverseGen，证明我们生成的数据既高效又多样。使用 ReverseGen 生成的数据进行微调的模型始终优于在人工标注数据或一般模型生成数据上训练的模型，为任务特定的大语言模型增强提供了一种新的数据合成视角。
英文摘要：
Large language models (LLMs) have significantly benefited from training on diverse, high-quality task-specific data, leading to impressive performance across a range of downstream applications. Current methods often rely on human-annotated data or predefined task templates to direct powerful LLMs in synthesizing task-relevant data for effective model training. However, this dependence on manually designed components may constrain the scope of generated data, potentially overlooking critical edge cases or novel scenarios that could challenge the model. In this paper, we present a novel approach, ReverseGen, designed to automatically generate effective training samples that expose the weaknesses of LLMs. Specifically, we introduce a dedicated proposer trained to produce queries that lead target models to generate unsatisfactory responses. These failure-inducing queries are then used to construct training data, helping to address the models' shortcomings and improve overall performance. Our approach is flexible and can be applied to models of various scales (3B, 7B, and 8B). We evaluate ReverseGen on three key applications (safety, honesty, and math), demonstrating that our generated data is both highly effective and diverse. Models fine-tuned with ReverseGen-generated data consistently outperform those trained on human-annotated or general model-generated data, offering a new perspective on data synthesis for task-specific LLM enhancement.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
