基于LLM导师优化对话级结果的强化学习效率方法
Efficient RL for optimizing conversation level outcomes with an LLM-based tutor
AI总结：
本文提出用低维潜在状态表示对话历史并优化长期策略来改进LLM数学辅导导师，以轻量级方法提升学生独立解题的长期效果。
摘要：
基于现有带人类反馈的强化学习（RLHF）框架构建的大语言模型（LLMs）通常根据即时轮次级人类偏好来优化响应。然而，这种方法在多轮对话场景中（如在线数学辅导）存在不足。我们提出一种方法，通过用学生的低维潜在状态表示来表征对话历史，并基于该潜在状态优化长期策略以确定高层动作，从而增强基于LLM的导师。其目标是使导师的行为更好地与长期目标对齐，即引导学生独立解决目标数学问题。我们的模型是轻量级的，与先前端到端训练导师策略以直接输出导师下一句话语的工作相比，所需的计算资源更少。我们的实验结果表明，在LLM模拟的辅导任务中，与提示方法相比，这些改进带来了更好的长期结果。
英文摘要：
Large language models (LLMs) built on existing reinforcement learning with human feedback (RLHF) frameworks typically optimize responses based on immediate turn-level human preferences. However, this approach falls short in multi-turn dialogue settings, such as online math tutoring. We propose a method to enhance LLM-based tutors by representing the dialogue history with a lower-dimensional latent state representation of a student and optimizing a long-term policy to determine high-level actions based on the latent state. The goal is to better align the tutor's behavior with the long-term objective of guiding the student towards solving a target math problem on their own. Our model is lightweight, requiring less computational resources than prior work of training the tutor policy end-to-end to directly output the tutor's next utterance. Our experiment results demonstrate that these modifications lead to improved long-term outcomes compared to prompting in LLM-simulated tutoring tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
