UltraLogic: 通过大规模数据合成和双极浮点奖励增强大语言模型推理
UltraLogic: Enhancing LLM Reasoning through Large-Scale Data Synthesis and Bipolar Float Reward
AI总结：
UltraLogic通过大规模数据合成和双极浮点奖励机制，提升大语言模型的推理能力，强调任务多样性和难度匹配对训练效率的促进作用。
摘要：
尽管大型语言模型（LLMs）在自然语言处理中展现了显著潜力，但需要多步骤逻辑、规划和验证的复杂通用推理仍然是关键瓶颈。尽管可验证奖励的强化学习（RLVR）在特定领域取得了成功，但该领域缺乏大规模、高质量且难度校准的数据用于通用推理。为此，我们提出了UltraLogic，一种通过基于代码的求解方法将问题的逻辑核心与其自然语言表达解耦的框架，以自动化高质量数据的生成。该框架包含数百种独特的任务类型，并在十个难度级别上具有自动校准管道。此外，为缓解二元奖励稀疏性和非负奖励陷阱，我们引入了双极浮点奖励（BFR）机制，利用分级惩罚有效区分完美响应与存在逻辑缺陷的响应。我们的实验表明，任务多样性是推理增强的主要驱动因素，且BFR结合难度匹配策略显著提高了训练效率，引导模型朝向全局逻辑最优解。
英文摘要：
While Large Language Models (LLMs) have demonstrated significant potential in natural language processing , complex general-purpose reasoning requiring multi-step logic, planning, and verification remains a critical bottleneck. Although Reinforcement Learning with Verifiable Rewards (RLVR) has succeeded in specific domains , the field lacks large-scale, high-quality, and difficulty-calibrated data for general reasoning. To address this, we propose UltraLogic, a framework that decouples the logical core of a problem from its natural language expression through a Code-based Solving methodology to automate high-quality data production. The framework comprises hundreds of unique task types and an automated calibration pipeline across ten difficulty levels. Furthermore, to mitigate binary reward sparsity and the Non-negative Reward Trap, we introduce the Bipolar Float Reward (BFR) mechanism, utilizing graded penalties to effectively distinguish perfect responses from those with logical flaws. Our experiments demonstrate that task diversity is the primary driver for reasoning enhancement , and that BFR, combined with a difficulty matching strategy, significantly improves training efficiency, guiding models toward global logical optima.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
