基于网页重构的从零开始指令微调数据合成
Instruction-Tuning Data Synthesis from Scratch via Web Reconstruction
AI总结：
针对现有指令微调数据合成方法依赖种子数据或网页结构强假设的问题，提出双视角网页重构框架WebR，可从原始网页自动合成高质量数据，在四个基准上最优提升16.65%，兼具兼容性、效率与可扩展性。
摘要：
大语言模型指令遵循能力的提升，关键取决于高质量指令-响应对的可用性。现有自动数据合成方法虽减轻了人工整理的负担，但往往严重依赖种子数据的质量，或是对网页文档的结构与内容存在强假设。为解决这些挑战，我们提出网页重构（Web Reconstruction, WebR），这是一个完全自动化的框架，可在极少假设下直接从原始网页文档合成高质量指令微调（instruction-tuning, IT）数据。我们利用原始网页内容的内在多样性，通过一种新颖的双视角范式——“网页即指令”与“网页即响应”——将网页重构概念化为指令微调数据合成任务，其中每个网页文档被指定为指令或响应，以启动重构流程。综合实验表明，WebR生成的数据集在四个指令遵循基准上优于当前最优基线，最高提升达16.65%。值得注意的是，WebR展现出优异的兼容性、数据效率与可扩展性，能够以极少的工作量增强领域适配能力。数据与代码已公开于https://github.com/YJiangcm/WebR。
英文摘要：
The improvement of LLMs' instruction-following capabilities depends critically on the availability of high-quality instruction-response pairs. While existing automatic data synthetic methods alleviate the burden of manual curation, they often rely heavily on either the quality of seed data or strong assumptions about the structure and content of web documents. To tackle these challenges, we propose Web Reconstruction (WebR), a fully automated framework for synthesizing high-quality instruction-tuning (IT) data directly from raw web documents with minimal assumptions. Leveraging the inherent diversity of raw web content, we conceptualize web reconstruction as an instruction-tuning data synthesis task via a novel dual-perspective paradigm--Web as Instruction and Web as Response--where each web document is designated as either an instruction or a response to trigger the reconstruction process. Comprehensive experiments show that datasets generated by WebR outperform state-of-the-art baselines by up to 16.65% across four instruction-following benchmarks. Notably, WebR demonstrates superior compatibility, data efficiency, and scalability, enabling enhanced domain adaptation with minimal effort. The data and code are publicly available at https://github.com/YJiangcm/WebR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
