LLM See, LLM Do: 引导数据生成以定向非可微目标
LLM See, LLM Do: Guiding Data Generation to Target Non-Differentiable Objectives
AI总结：
本文提出“主动继承”方法，通过按非可微目标约束合成数据，引导大语言模型在测试时生成具有期望属性（如高词汇多样性或低毒性）的内容。
摘要：
合成数据的广泛采用引发了新的问题，即生成数据的模型如何通过蒸馏数据影响其他大型语言模型（LLM）。首先，我们的工作通过系统研究合成数据整合的后果，详尽地刻画了模型属性被动继承的影响。我们提供了迄今为止最全面的研究之一，探讨合成数据的来源如何塑造模型的内部偏见、校准以及生成文本的属性和偏好。我们发现，即使合成数据提示看似“中性”，模型对某些属性也表现出惊人的敏感性，这引出了一个疑问：这种敏感性能否被善加利用。\n\n我们的发现进一步提出了一个问题：我们能否通过利用数据生成过程，在测试时显式地将模型引导至我们期望的属性？由于收集具有特定特征或目标的数据成本高昂，这在历史上被认为不可行。然而，合成数据质量的提升，以及向旨在遵循多样化指令的通用模型的转变，使得这个问题变得适时。我们提出“主动继承”这一术语，用以描述根据非可微目标有意约束合成数据的行为。我们展示了主动继承如何将模型的生成特征引导至理想的非可微属性，例如高词汇多样性或低毒性。
英文摘要：
The widespread adoption of synthetic data raises new questions about how models generating the data can influence other large language models (LLMs) via distilled data. To start, our work exhaustively characterizes the impact of passive inheritance of model properties by systematically studying the consequences of synthetic data integration. We provide one of the most comprehensive studies to-date of how the source of synthetic data shapes models' internal biases, calibration and generations' textual attributes and preferences. We find that models are surprisingly sensitive towards certain attributes even when the synthetic data prompts appear "neutral". which invites the question whether this sensitivity can be exploited for good. Our findings invite the question can we explicitly steer the models towards the properties we want at test time by exploiting the data generation process? This would have historically been considered infeasible due to the cost of collecting data with a specific characteristic or objective in mind. However, improvement in the quality of synthetic data, as well as a shift towards general-purpose models designed to follow a diverse way of instructions, means this question is timely. We propose active inheritance as a term to describe intentionally constraining synthetic data according to a non-differentiable objective. We demonstrate how active inheritance can steer the generation profiles of models towards desirable non-differentiable attributes, e.g. high lexical diversity or low toxicity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
