WildLong：大规模合成真实长上下文指令数据
WildLong: Synthesizing Realistic Long-Context Instruction Data at Scale
AI总结：
WildLong通过提取真实查询元信息、图建模共现关系和自适应生成，合成多样化长上下文指令数据，微调模型在基准上超越现有开源长上下文模型，并保持短上下文性能。
摘要：
具有扩展上下文窗口的大型语言模型（LLMs）能够处理需要广泛信息整合的任务，但受限于高质量、多样化长上下文指令调优数据集的稀缺性。现有的数据合成方法聚焦于事实检索和摘要等狭窄目标，限制了其对复杂真实世界任务的泛化能力。WildLong从真实用户查询中提取元信息，通过基于图的方法建模共现关系，并采用自适应生成来产生可扩展的数据。它超越了单文档任务，支持多文档推理，如跨文档比较和聚合。我们使用WildLong合成的150K指令-响应对微调的模型，在基准测试中超越了现有的开源长上下文优化模型，同时在没有补充短上下文数据的情况下，保持了短上下文任务的强性能。通过生成更加多样化和真实的长上下文指令数据集，WildLong增强了LLMs在长上下文中泛化到复杂真实世界推理的能力，为长上下文数据合成建立了新范式。
英文摘要：
Large language models (LLMs) with extended context windows enable tasks requiring extensive information integration but are limited by the scarcity of high-quality, diverse datasets for long-context instruction tuning. Existing data synthesis methods focus narrowly on objectives like fact retrieval and summarization, restricting their generalizability to complex, real-world tasks. WildLong extracts meta-information from real user queries, models co-occurrence relationships via graph-based methods, and employs adaptive generation to produce scalable data. It extends beyond single-document tasks to support multi-document reasoning, such as cross-document comparison and aggregation. Our models, finetuned on 150K instruction-response pairs synthesized using WildLong, surpasses existing open-source long-context-optimized models across benchmarks while maintaining strong performance on short-context tasks without incorporating supplementary short-context data. By generating a more diverse and realistic long-context instruction dataset, WildLong enhances LLMs' ability to generalize to complex, real-world reasoning over long contexts, establishing a new paradigm for long-context data synthesis.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
