PRefLexOR：基于偏好的递归语言建模用于推理与智能体思维的探索性优化
PRefLexOR: Preference-based Recursive Language Modeling for Exploratory Optimization of Reasoning and Agentic Thinking
AI总结：
PRefLexOR提出一种基于偏好的递归语言建模方法，结合强化学习与检索增强知识图谱，使仅30亿参数的小模型通过多阶段迭代推理自我教学，在生物材料科学等跨域任务中实现更深层推理与自适应改进。
摘要：
PRefLexOR（基于偏好的递归语言建模用于推理的探索性优化）将偏好优化与强化学习中的概念相结合，使模型能够通过迭代推理改进实现自我教学。我们提出了一种递归学习方法，在训练和推理阶段让模型参与多步推理、回访和精炼中间步骤，然后再产生最终输出。通过多个训练阶段，模型首先通过优化偏好响应与非偏好响应之间的对数几率，学习使其推理与正确的决策路径对齐。在此过程中，PRefLexOR通过从随机文本块中生成问题并进行检索增强来构建动态知识图谱，从而将整个训练语料库中的相关细节进行上下文化。在第二阶段，偏好优化通过使用拒绝采样来微调推理质量，同时持续生成原位训练数据并掩蔽推理步骤，从而提升模型性能。在思考令牌框架内的递归优化引入了迭代反馈循环，模型在其中精炼推理，实现更深层的连贯性、一致性和适应性。我们在仅含30亿参数的小型语言模型中实现了该方法，结果表明即使是微型模型也能迭代地自我教学，以更高的深度和反思性进行推理。我们的实现简单直接，可以整合到任何现有的预训练大语言模型中。我们将示例聚焦于生物材料科学的应用，并在从域内到跨域应用的各种案例研究中展示了该方法。利用包含思考和反思模态的推理策略，我们构建了一种多智能体递归自我改进推理方法，通过在推理时间内重复采样来逐步改进响应。
英文摘要：
PRefLexOR (Preference-based Recursive Language Modeling for Exploratory Optimization of Reasoning) combines preference optimization with concepts from Reinforcement Learning to enable models to self-teach through iterative reasoning improvements. We propose a recursive learning approach that engages the model in multi-step reasoning, revisiting, and refining intermediate steps before producing a final output in training and inference phases. Through multiple training stages, the model first learns to align its reasoning with accurate decision paths by optimizing the log odds between preferred and non-preferred responses. During this process, PRefLexOR builds a dynamic knowledge graph by generating questions from random text chunks and retrieval-augmentation to contextualize relevant details from the entire training corpus. In the second stage, preference optimization enhances model performance by using rejection sampling to fine-tune reasoning quality by continually producing in-situ training data while masking the reasoning steps. Recursive optimization within a thinking token framework introduces iterative feedback loops, where the model refines reasoning, achieving deeper coherence, consistency, and adaptability. Implemented in small language models with only 3 billion parameters, we should that even tiny models can iteratively teach themselves to reason with greater depth and reflectivity. Our implementation is straightforward and can be incorporated into any existing pretrained LLM. We focus our examples on applications in biological materials science and demonstrate the method in a variety of case studies that range from in-domain to cross-domain applications. Using reasoning strategies that include thinking and reflection modalities we build a multi-agent recursive self-improving inference approach to successively improve responses via repeated sampling in inference time.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
