通过多智能体模拟合成大语言模型的后训练数据
Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation
AI总结：
本文提出多智能体模拟器MATRIX及场景驱动生成器MATRIX-Gen，以可扩展方式合成高质量后训练指令数据，仅2万对数据即让Llama-3-8B-Base在两项基准上超越Llama-3-8B-Instruct。
摘要：
后训练对于使大语言模型（LLM）能够遵循人类指令至关重要。然而，其有效性依赖于高质量的指令数据，而由于隐私问题、数据稀缺和高昂的标注成本，这类数据在现实世界中难以获取。为填补这一空白，受近期利用LLM模拟人类社会的成功启发，我们提出了MATRIX，这是一个多智能体模拟器，能够自动生成多样化的基于文本的场景，以真实且可扩展的方式捕捉广泛的现实人类需求。利用这些输出，我们引入了一种新颖的场景驱动指令生成器MATRIX-Gen，用于可控且高度真实的数据合成。大量实验表明，我们的框架能够有效生成通用数据和特定领域数据。在AlpacaEval 2和Arena-Hard基准测试中，仅使用MATRIX-Gen合成的2万条指令-响应对进行后训练的Llama-3-8B-Base，表现优于在超过1000万对数据上训练的Meta Llama-3-8B-Instruct模型。
英文摘要：
Post-training is essential for enabling large language models (LLMs) to follow human instructions. However, its effectiveness depends on high-quality instruction data, which is challenging to obtain in the real world due to privacy concerns, data scarcity, and high annotation costs. To fill this gap, inspired by the recent success of using LLMs to simulate human society, we propose MATRIX, a multi-agent simulator that automatically generates diverse text-based scenarios, capturing a wide range of real-world human needs in a realistic and scalable manner. Leveraging these outputs, we introduce a novel scenario-driven instruction generator MATRIX-Gen for controllable and highly realistic data synthesis. Extensive experiments demonstrate that our framework effectively generates both general and domain-specific data. On AlpacaEval 2 and Arena-Hard benchmarks, Llama-3-8B-Base, post-trained on datasets synthesized by MATRIX-Gen with just 20K instruction-response pairs, outperforms Meta's Llama-3-8B-Instruct model, which was trained on over 10M pairs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
