arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.17461cs.CLcs.AI
收藏

利用大语言模型智能体生成合成对话数据集

Synthetic Dialogue Dataset Generation using LLM Agents

Macquarie University(麦考瑞大学);Deakin University(迪肯大学);The University of Melbourne(墨尔本大学)

Yelaman Abdullin, Diego Molla-Aliod, Bahadorreza Ofoghi, John Yearwood, Qingyang Li

AI总结:

针对普通用户难以为自身问题构建线性规划模型的痛点,提出利用两个LLM智能体分别模拟用户与对话助手生成合成对话数据集的方法,经多维度评估验证对话质量良好,相关资源与基线模型已对外开放。

摘要:

线性规划(LP)问题在现实应用中普遍存在。然而,尽管这类问题看似简单,未经训练的用户可能很难为自己的具体问题确定线性模型。我们设想构建一个面向目标的对话智能体,它可以与用户进行对话,获取所有必要信息,以便后续的智能体能够生成线性模型。本文提出了一种生成样本对话的方法,这些样本对话可用于开发和训练此类对话智能体。我们通过提示工程开发了两个相互“对话”的智能体,一个扮演对话智能体,另一个扮演用户。我们使用仅提供给用户的NL4Opt数据集中的线性问题文本描述集合,让智能体和用户进行对话,直到智能体从原始问题描述中获取所有关键信息为止。我们还提出了一种对话的外部评估方法,即评估由对话生成的摘要与原始问题描述的匹配程度。我们开展了人工评估和自动评估,其中包括一种使用GPT-4模拟人工评估指标的评估方法。评估结果表明,对话整体质量良好,不过仍需开展研究以提升GPT-4评估指标的质量。生成的对话(包括其中一个子集的人工标注结果)已向研究界开放。用于生成这些对话的对话智能体可作为基线模型。

英文摘要:

Linear programming (LP) problems are pervasive in real-life applications. However, despite their apparent simplicity, an untrained user may find it difficult to determine the linear model of their specific problem. We envisage the creation of a goal-oriented conversational agent that will engage in conversation with the user to elicit all information required so that a subsequent agent can generate the linear model. In this paper, we present an approach for the generation of sample dialogues that can be used to develop and train such a conversational agent. Using prompt engineering, we develop two agents that "talk" to each other, one acting as the conversational agent, and the other acting as the user. Using a set of text descriptions of linear problems from NL4Opt available to the user only, the agent and the user engage in conversation until the agent has retrieved all key information from the original problem description. We also propose an extrinsic evaluation of the dialogues by assessing how well the summaries generated by the dialogues match the original problem descriptions. We conduct human and automatic evaluations, including an evaluation approach that uses GPT-4 to mimic the human evaluation metrics. The evaluation results show an overall good quality of the dialogues, though research is still needed to improve the quality of the GPT-4 evaluation metrics. The resulting dialogues, including the human annotations of a subset, are available to the research community. The conversational agent used for the generation of the dialogues can be used as a baseline.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑