在JSON内部思考：用于大语言模型严格模式遵循的强化策略
Think Inside the JSON: Reinforcement Strategy for Strict LLM Schema Adherence
AI总结：
本研究针对大语言模型生成的严格模式遵循问题，基于DeepSeek R1框架结合合成数据集与GRPO定制奖励训练1.5B参数模型，经小规模训练后在模式一致性上优于多款大参数量模型，验证了资源高效模式约束生成框架的实用性。
摘要：
本文旨在解决通过利用大语言模型（LLM）的推理能力，在其生成过程中强制执行严格模式遵循的挑战。我们的方法基于DeepSeek R1强化学习框架，通过一套新颖的流水线训练参数量为1.5B的模型的结构化推理能力，该流水线结合了合成推理数据集构建与组相对策略优化（GRPO）下的定制奖励函数。具体而言，我们首先在一个包含2万样本的非结构化转结构化数据集上开展R1强化学习，复刻原始DeepSeek R1方法以建立核心推理能力。随后，我们在另一个包含1万推理样本的数据集上进行有监督微调（SFT），重点优化下游任务的模式遵循效果。尽管训练规模相对有限——GRPO训练在8卡H100 GPU集群上耗时约20小时，SFT在单卡A100上耗时3小时——我们的模型在强制执行模式一致性方面展现出稳健性能。我们将ThinkJSON方法与原始DeepSeek R1（671B）、DeepSeek R1的蒸馏版本（Qwen-1.5B和Qwen-7B）以及Gemini 2.0 Flash（70B）进行对比，证明了其在实际应用中的有效性。我们的结果凸显了资源高效的模式约束文本生成框架的实用价值。
英文摘要：
In this paper, we address the challenge of enforcing strict schema adherence in large language model (LLM) generation by leveraging LLM reasoning capabilities. Building on the DeepSeek R1 reinforcement learning framework, our approach trains structured reasoning skills of a 1.5B parameter model through a novel pipeline that combines synthetic reasoning dataset construction with custom reward functions under Group Relative Policy Optimization (GRPO). Specifically, we first perform R1 reinforcement learning on a 20K sample unstructured-to-structured dataset, mirroring the original DeepSeek R1 methods, to establish core reasoning abilities. Subsequently, we performed supervised fine-tuning on a separate 10K reasoning sample dataset, focusing on refining schema adherence for downstream tasks. Despite the relatively modest training scope, requiring approximately 20 hours on an 8xH100 GPU cluster for GRPO training and 3 hours on 1xA100 for SFT, our model demonstrates robust performance in enforcing schema consistency. We compare our ThinkJSON approach against the original DeepSeek R1 (671B), distilled versions of DeepSeek R1 (Qwen-1.5B and Qwen-7B), and Gemini 2.0 Flash (70B), showcasing its effectiveness in real-world applications. Our results underscore the practical utility of a resource-efficient framework for schema-constrained text generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
