发表机构
The Ohio State University（俄亥俄州立大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究揭示生成模型自消耗训练中用户偏好与模型分布共同演化的动态，证明纯合成数据导致偏差放大，而注入参考数据可实现唯一全局均衡，并提出联合优化参考分布与混合权重的控制算法。
AI 中文摘要
生成模型日益在自消耗的迭代循环中训练，其中用户从模型生成的候选中策展偏好样本，而策展样本被用于训练下一代模型。先前的工作大多假设用户偏好是固定的，但在实践中，接触模型输出会逐渐重塑用户认为理想的内容，从而形成一个模型分布与用户偏好共同演化的反馈循环。我们迈出理解这种耦合动态长期行为的第一步。我们证明，当训练完全依赖于用户策展的合成数据时，迭代策展会放大初始偏差，并将系统推向多个单例均衡之一，在这些均衡中，具有初始优势的实例最终占据主导地位。相反，以足够大的比例将参考数据注入训练中，会从根本上改变动态，并产生一个唯一的全局吸引均衡。基于这一见解，我们研究了如何利用参考数据注入来控制长期结果，并提出了一种高效算法，该算法联合选择参考分布及其混合权重，以将耦合系统引导至保留所需属性同时最小化数据收集成本的均衡。
英文摘要
Generative models are increasingly trained in self-consuming iterative loops, where users curate preferred samples from model-generated candidates and the curated samples are used to train future generations of the model. Prior work has largely assumed fixed user preferences, but in practice exposure to model outputs gradually reshapes what users perceive as desirable, creating a feedback loop in which model distributions and user preferences co-evolve. We take a first step toward understanding the long-term behavior of such coupled dynamics. We show that when training relies entirely on user-curated synthetic data, iterative curation amplifies initial biases and drives the system toward one of multiple singleton equilibria in which the instance holding an initial advantage eventually dominates. In contrast, injecting reference data into training at a sufficiently large rate fundamentally changes the dynamics and yields a unique globally attracting equilibrium. Building on this insight, we study how reference-data injection can be used to control long-term outcomes, and propose an efficient algorithm that jointly selects a reference distribution and its mixing weight to steer the coupled system toward equilibria that preserve desired attributes while minimizing data collection costs.
CommentsPublished as a conference paper at NeurIPS 2026
