通过符号反馈进行LLM微调：RLSF
RLSF: Fine-tuning LLMs via Symbolic Feedback
AI总结：
RLSF通过符号反馈微调LLM，利用符号推理工具提供精细反馈，提升领域特定任务的性能，使小模型超越大模型。
摘要：
大型语言模型（LLMs）已经改变了人工智能，但它们在需要领域特定推理和逻辑对齐的任务上往往表现不佳。传统微调方法没有利用通过符号推理工具（例如证明器）可用的大量符号领域知识，并且进一步受到稀疏奖励和不可靠的奖励模型的限制。我们引入了通过符号反馈进行强化学习（RLSF），一种新的微调范式，其中符号推理工具（例如求解器、证明器和代数系统）为LLMs提供精细反馈。RLSF使用由符号工具生成的多大小证书（例如证明）来识别和纠正模型输出中的错误，提供令牌级别的指导，而无需可微推理系统。这种范式弥合了符号推理和LLM微调之间的差距，使模型能够与领域特定约束精确对齐，同时解决传统奖励信号的关键限制。通过广泛的评估，我们证明了基于RLSF的LLM微调在五个不同的应用中优于传统方法（这些应用具有某些关联的逻辑或领域约束），即从自然语言伪代码到编程语言的程序合成，三个化学任务，以及解决24点游戏。一个关键结论是，通过RLSF进行微调使相对较小的LLM能够显著超越比它们大数个数量级的闭源模型。
英文摘要：
Large Language Models (LLMs) have transformed AI but often struggle with tasks that require domain-specific reasoning and logical alignment. Traditional fine-tuning methods do not leverage the vast amount of symbolic domain-knowledge available to us via symbolic reasoning tools (e.g., provers), and are further limited by sparse rewards and unreliable reward models. We introduce Reinforcement Learning via Symbolic Feedback (RLSF), a novel fine-tuning paradigm where symbolic reasoning tools (e.g., solvers, provers, and algebra systems) provide fine-grained feedback to LLMs. RLSF uses poly-sized certificates (e.g., proofs) generated by symbolic tools to identify and correct errors in model outputs, offering token-level guidance without requiring differentiable reasoning systems. This paradigm bridges the gap between symbolic reasoning and LLM fine-tuning, enabling precise alignment with domain-specific constraints while addressing key limitations of traditional reward signals. Via extensive evaluations, we show that our RLSF-based fine-tuning of LLMs outperforms traditional approaches on five different applications (that have some associated logical or domain constraints), namely, program synthesis from natural language pseudo-code to programming language, three chemistry tasks, and solving the Game of 24. A key takeaway is that fine-tuning via RLSF enables relatively smaller LLMs to significantly outperform closed-source models that are orders of magnitude larger.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
