多模态合成数据训练与模型崩溃：来自 VLM 与扩散模型的洞见
Multi-modal Synthetic Data Training and Model Collapse: Insights from VLMs and Diffusion Models
AI总结：
本文将模型崩溃研究扩展到 VLM、文本到图像扩散模型及多模型递归训练循环，揭示其多模态特征，并验证多种缓解策略。
摘要：
近期研究强调了生成式模型崩溃的风险：当模型持续在自生成数据上训练时，其性能会逐步退化。然而，现有关于模型崩溃的探索仅限于单一、单模态模型，这限制了我们在更现实场景中的理解，例如多样化的多模态 AI 智能体通过合成数据自主交互并持续演化。我们将合成数据训练与模型崩溃研究扩展到多模态视觉—语言生成系统，如视觉—语言模型（VLM）和文本到图像扩散模型，以及涉及多个模型的递归“生成—训练”循环。我们发现，此前在单模态生成模型中观察到的模型崩溃，在多模态情境下表现出不同特征，例如在 VLM 图像描述任务中，视觉—语言对齐有所改善，但方差增大。此外，我们发现增加解码预算、提高模型多样性以及使用冻结模型重新标注等通用方法，能够有效缓解模型崩溃。我们的发现为降低自我改进的多智能体 AI 系统中的模型崩溃风险，以及构建稳健的多模态合成数据集，提供了初步洞见和实践指南。
英文摘要：
Recent research has highlighted the risk of generative model collapse, where performance progressively degrades when continually trained on self-generated data. However, existing exploration on model collapse is limited to single, unimodal models, limiting our understanding in more realistic scenarios, such as diverse multi-modal AI agents interacting autonomously through synthetic data and continually evolving. We expand the synthetic data training and model collapse study to multi-modal vision-language generative systems, such as vision-language models (VLMs) and text-to-image diffusion models, as well as recursive generate-train loops with multiple models. We find that model collapse, previously observed in single-modality generative models, exhibits distinct characteristics in the multi-modal context, such as improved vision-language alignment and increased variance in VLM image-captioning task. Additionally, we find that general approaches such as increased decoding budgets, greater model diversity, and relabeling with frozen models can effectively mitigate model collapse. Our findings provide initial insights and practical guidelines for reducing the risk of model collapse in self-improving multi-agent AI systems and curating robust multi-modal synthetic datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
