LongMagpie：一种用于生成大规模长上下文指令的自合成方法
LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions
AI总结：
LongMagpie利用对齐后的长上下文LLMs自回归生成文档相关查询，自动合成大规模高质量长上下文指令数据，在多个基准上取得领先性能。
摘要：
高质量的长上下文指令数据对于对齐长上下文大语言模型（LLMs）至关重要。尽管Qwen和Llama等模型已公开发布，但它们的长上下文指令数据仍为专有。人工标注成本高昂且具有挑战性，而基于模板的合成方法限制了规模、多样性和质量。我们提出LongMagpie，一个自动生成大规模长上下文指令数据的自合成框架。我们的核心洞见是：对齐后的长上下文LLMs在给定一篇文档、后接用户轮次之前的特殊标记时，会自回归地生成与上下文相关的查询。通过收集这些文档-查询对以及模型的响应，LongMagpie无需人工即可生成高质量指令。在HELMET、RULER和Longbench v2上的实验表明，LongMagpie在长上下文任务上取得了领先性能，同时在短上下文任务上保持了有竞争力的表现，确立了其作为一种简单有效的方法，用于开放、多样且可扩展的长上下文指令数据合成。
英文摘要：
High-quality long-context instruction data is essential for aligning long-context large language models (LLMs). Despite the public release of models like Qwen and Llama, their long-context instruction data remains proprietary. Human annotation is costly and challenging, while template-based synthesis methods limit scale, diversity, and quality. We introduce LongMagpie, a self-synthesis framework that automatically generates large-scale long-context instruction data. Our key insight is that aligned long-context LLMs, when presented with a document followed by special tokens preceding a user turn, auto-regressively generate contextually relevant queries. By harvesting these document-query pairs and the model's responses, LongMagpie produces high-quality instructions without human effort. Experiments on HELMET, RULER, and Longbench v2 demonstrate that LongMagpie achieves leading performance on long-context tasks while maintaining competitive performance on short-context tasks, establishing it as a simple and effective approach for open, diverse, and scalable long-context instruction data synthesis.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
