PuzzleClone: 一种基于DSL的可验证数据合成框架
PuzzleClone: A DSL-Powered Framework for Synthesizing Verifiable Data
AI总结：
提出PuzzleClone框架，通过DSL驱动的方法合成大规模、高可靠、多样化的可验证数学逻辑数据集，并构建PC-83K基准，实验表明后训练能显著提升LLM在逻辑与数学任务上的性能。
摘要：
高质量、带有可验证答案的数学和逻辑数据集对于增强大型语言模型（LLM）的推理能力至关重要。虽然最近的数据增强技术促进了大规模基准的创建，但现有的LLM生成数据集往往存在可靠性、多样性和可扩展性有限的问题。为了解决这些挑战，我们引入了PuzzleClone，一个使用新颖的DSL驱动方法大规模合成可验证数据的正式框架。我们的方法具有三个关键创新：（1）将种子谜题编码为结构化的逻辑规范，（2）通过系统化的变量和约束随机化生成可扩展的变体，（3）通过再现机制确保有效性。应用PuzzleClone，我们构建了PC-83K，一个包含超过83K个多样化且经过程序验证的谜题的基准。生成的谜题涵盖了广泛的难度和格式，对当前最先进的模型构成了重大挑战。实验结果表明，在PC-83K上进行后训练（SFT和RL）不仅在测试集上取得了显著提升，而且在各种逻辑和数学基准上也取得了改进。后训练将PC-83K上的平均性能从14.5提高到66.0，并在7个逻辑和数学基准上持续改进，绝对百分点最高达18.4（SATBench从51.6提高到70.0）。我们的代码和数据可在https://github.com/HiThink-Research/PuzzleClone获取。
英文摘要：
High-quality mathematical and logical datasets with verifiable answers are essential for strengthening the reasoning capabilities of large language models (LLMs). While recent data augmentation techniques have facilitated the creation of large-scale benchmarks, existing LLM-generated datasets often suffer from limited reliability, diversity, and scalability. To address these challenges, we introduce PuzzleClone, a formal framework for synthesizing verifiable data at scale using a novel DSL-driven approach. Our approach features three key innovations: (1) encoding seed puzzles into structured logical specifications, (2) generating scalable variants through systematic variable and constraint randomization, and (3) ensuring validity via a reproduction mechanism. Applying PuzzleClone, we construct PC-83K, a benchmark comprising over 83K diverse and programmatically validated puzzles. The generated puzzles span a wide spectrum of difficulty and formats, posing significant challenges to current state-of-the-art models. Experimental results show that post training (SFT and RL) on PC-83K yields substantial improvements not only on the testset but also on various logic and mathematical benchmarks. Post training raises average performance on PC-83K from 14.5 to 66.0 and delivers consistent improvements across 7 logic and mathematical benchmarks up to 18.4 absolute percentage points (SATBench from 51.6 to 70.0). Our code and data are available at https://github.com/HiThink-Research/PuzzleClone.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
