LLM 应当像人类一样思考与行动
LLM should think and action as a human
AI总结：
针对大语言模型多轮对话易出错、难适配不同需求、工具调用低效等问题，提出基于内置思维链的思考方法，通过监督微调与强化学习增强模型推理规划能力，解决多轮对话相关问题。
摘要：
近来将大语言模型训练为聊天助手的做法十分流行，但在用户与聊天助手的对话中存在提示词，需要聊天助手与用户进行多轮交互。然而多轮对话存在诸多问题：聊天助手的回复容易出错，无法帮助用户达成目标，且随着对话轮次增加，出错概率也会上升；聊天助手难以针对同一提示词根据实际需求生成带有不同流程的回复；聊天助手需要使用工具，但当前的工具调用方式不够优雅高效，且调用次数受限。这些问题的主要原因是大语言模型不具备人类的思考能力，缺乏推理能力、规划能力以及执行计划的能力。为解决这些问题，我们提出一种基于内置思维链的思考方法：在多轮对话中，针对每一条用户提示词，大语言模型基于聊天历史、思考上下文、动作调用、记忆与知识等要素进行思考，开展详细的推理与规划，并按照规划执行动作。我们还探究了大语言模型如何通过该思考方法增强思考能力：依据该思考方法收集训练数据集，通过监督学习对大语言模型进行微调；训练一个一致性奖励模型，并将其作为奖励函数，使用强化学习对大语言模型进行微调，经过强化的大语言模型会按照这种思考方式输出结果。我们的实验结果表明，大语言模型的推理能力与规划能力得到增强，多轮对话中存在的问题也得到了解决。
英文摘要：
It is popular lately to train large language models to be used as chat assistants, but in the conversation between the user and the chat assistant, there are prompts, require multi-turns between the chat assistant and the user. However, there are a number of issues with the multi-turns conversation: The response of the chat assistant is prone to errors and can't help users achieve their goals, and as the number of conversation turns increases, the probability of errors will also increase; It is difficult for chat assistant to generate responses with different processes based on actual needs for the same prompt; Chat assistant require the use of tools, but the current approach is not elegant and efficient, and the number of tool calls is limited. The main reason for these issues is that large language models don't have the thinking ability as a human, lack the reasoning ability and planning ability, and lack the ability to execute plans. To solve these issues, we propose a thinking method based on a built-in chain of thought: In the multi-turns conversation, for each user prompt, the large language model thinks based on elements such as chat history, thinking context, action calls, memory and knowledge, makes detailed reasoning and planning, and actions according to the plan. We also explored how the large language model enhances thinking ability through this thinking method: Collect training datasets according to the thinking method and fine tune the large language model through supervised learning; Train a consistency reward model and use it as a reward function to fine tune the large language model using reinforcement learning, and the reinforced large language model outputs according to this way of thinking. Our experimental results show that the reasoning ability and planning ability of the large language model are enhanced, and the issues in the multi-turns conversation are solved.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
