arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.04286cs.AIcs.CL
收藏

使用大型语言模型生成真实的多智能体知识工作数据集

Using Large Language Models to Generate Authentic Multi-agent Knowledge Work Datasets

German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心);University of Kaiserslautern-Landau (RPTU)(凯泽斯劳滕-兰道大学);RheinMain University of Applied Sciences(莱茵美因应用科学大学)

Desiree Heim, Christian Jilek, Adrian Ulges, Andreas Dengel

AI总结:

针对真实知识工作数据难以收集的问题,本文提出可配置的多智能体LLM数据集生成器,通过模拟协作、记录知识图谱并开展人工真实性评估验证其潜力。

摘要:

当前公开可用的知识工作数据集合缺乏多样性、广泛的标注以及关于用户及其文档的上下文信息。这些问题阻碍了对知识工作辅助系统进行客观、可比较的数据驱动评估和优化。由于在真实场景中收集此类数据需要大量资源,并且必须进行数据审查,因此收集这样的数据集几乎是不可能的。为此,我们提出了一种可配置的多智能体知识工作数据集生成器。该系统模拟智能体之间的协作知识工作,生成由大型语言模型产生的文档及配套的数据轨迹。此外,生成器将其配置中给出的或在模拟过程中创建的所有背景信息捕获到知识图谱中。最终,所得数据集可以在不存在隐私或保密顾虑的情况下被使用和共享。本文介绍了我们方法的设计与愿景,并重点关注使用大型语言模型生成真实的知识工作文档。我们的研究中,人工评分者将53%的生成文档和74%的真实文档评定为逼真,这证明了我们方法的潜力。此外,我们分析了参与者评论中提到的真实性标准,并详细阐述了针对已识别共性问题的潜在改进。

英文摘要:

Current publicly available knowledge work data collections lack diversity, extensive annotations, and contextual information about the users and their documents. These issues hinder objective and comparable data-driven evaluations and optimizations of knowledge work assistance systems. Due to the considerable resources needed to collect such data in real-life settings and the necessity of data censorship, collecting such a dataset appears nearly impossible. For this reason, we propose a configurable, multi-agent knowledge work dataset generator. This system simulates collaborative knowledge work among agents producing Large Language Model-generated documents and accompanying data traces. Additionally, the generator captures all background information, given in its configuration or created during the simulation process, in a knowledge graph. Finally, the resulting dataset can be utilized and shared without privacy or confidentiality concerns. This paper introduces our approach's design and vision and focuses on generating authentic knowledge work documents using Large Language Models. Our study involving human raters who assessed 53% of the generated and 74% of the real documents as realistic demonstrates the potential of our approach. Furthermore, we analyze the authenticity criteria mentioned in the participants' comments and elaborate on potential improvements for identified common issues.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑