OCEAN：大语言模型中的离线思维链评估与对齐
OCEAN: Offline Chain-of-thought Evaluation and Alignment in Large Language Models
AI总结：
本研究针对大语言模型离线思维链评估不足的问题，提出OCEAN框架，通过知识图谱结合强化学习构建评估估计量并实现离策略优化，可高效提升思维链对齐效果且不损害模型通用能力。
摘要：
大语言模型（LLM）的离线评估对于理解其能力至关重要，不过现有研究中相关方法的探索仍不充分。本研究聚焦于思维链能力的离线评估，并展示如何基于所提出的评估方法优化大语言模型。为了实现具备丰富知识与推理路径的离线反馈，我们使用知识图谱（如Wikidata5m）为生成的思维链提供反馈。由于大语言模型推理与知识图谱结构之间存在异质性，知识图谱难以直接对大语言模型的行为进行交互与反馈，因为这需要对大语言模型生成的思维链进行精准的实体链接与知识图谱接地。为解决上述挑战，我们提出离线思维链评估框架OCEAN，该框架将大语言模型中的思维链推理建模为马尔可夫决策过程（MDP），并评估策略与知识图谱偏好建模的对齐程度。为克服推理异质性与接地问题，我们利用同策略知识图谱探索与强化学习（RL）建模知识图谱策略，该策略为大语言模型生成的思维链推理路径生成词元级似然分布，模拟知识图谱的推理偏好。随后，我们将知识图谱对生成推理路径的有效性与对齐性的反馈纳入逆倾向得分，提出KG-IPS估计量。理论上，我们证明了所提出的KG-IPS估计量的无偏性，并给出其方差的下界。借助离策略评估得到的值函数，我们可直接开展离策略优化，以进一步增强思维链对齐。实证研究表明，OCEAN能够被高效优化，生成具备更高估计值的思维链推理路径，同时不会影响大语言模型在下游任务中的通用能力或其内部知识。
英文摘要：
Offline evaluation of LLMs is crucial in understanding their capacities, though current methods remain underexplored in existing research. In this work, we focus on the offline evaluation of the chain-of-thought capabilities and show how to optimize LLMs based on the proposed evaluation method. To enable offline feedback with rich knowledge and reasoning paths, we use knowledge graphs (e.g., Wikidata5m) to provide feedback on the generated chain of thoughts. Due to the heterogeneity between LLM reasoning and KG structures, direct interaction and feedback from KGs on LLM behavior are challenging, as they require accurate entity linking and grounding of LLM-generated chains of thought in the KG. To address the above challenge, we propose an offline chain-of-thought evaluation framework, OCEAN, which models chain-of-thought reasoning in LLMs as an MDP and evaluate the policy's alignment with KG preference modeling. To overcome the reasoning heterogeneity and grounding problems, we leverage on-policy KG exploration and RL to model a KG policy that generates token-level likelihood distributions for LLM-generated chain-of-thought reasoning paths, simulating KG reasoning preference. Then we incorporate the knowledge-graph feedback on the validity and alignment of the generated reasoning paths into inverse propensity scores and propose KG-IPS estimator. Theoretically, we prove the unbiasedness of the proposed KG-IPS estimator and provide a lower bound on its variance. With the off-policy evaluated value function, we can directly enable off-policy optimization to further enhance chain-of-thought alignment. Our empirical study shows that OCEAN can be efficiently optimized for generating chain-of-thought reasoning paths with higher estimated values without affecting LLMs' general abilities in downstream tasks or their internal knowledge.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
