Using Large Language Models to Generate Authentic Multi-agent Knowledge Work Datasets
German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心);University of Kaiserslautern-Landau (RPTU)(凯泽斯劳滕-兰道大学);RheinMain University of Applied Sciences(莱茵美因应用科学大学)
Desiree Heim, Christian Jilek, Adrian Ulges, Andreas Dengel
备注:
Accepted and published (INFORMATIK Festival, Wiesbaden, 2024)
Current publicly available knowledge work data collections lack diversity, extensive annotations, and contextual information about the users and their documents. These issues hinder objective and comparable data-driven evaluations and optimizations of knowledge work assistance systems. Due to the considerable resources needed to collect such data in real-life settings and the necessity of data censorship, collecting such a dataset appears nearly impossible. For this reason, we propose a configurable, multi-agent knowledge work dataset generator. This system simulates collaborative knowledge work among agents producing Large Language Model-generated documents and accompanying data traces. Additionally, the generator captures all background information, given in its configuration or created during the simulation process, in a knowledge graph. Finally, the resulting dataset can be utilized and shared without privacy or confidentiality concerns.
This paper introduces our approach's design and vision and focuses on generating authentic knowledge work documents using Large Language Models. Our study involving human raters who assessed 53% of the generated and 74% of the real documents as realistic demonstrates the potential of our approach. Furthermore, we analyze the authenticity criteria mentioned in the participants' comments and elaborate on potential improvements for identified common issues.
Towards the "Digital Me": A vision of authentic Conversational Agents powered by personal Human Digital Twins
Technical University of Denmark(丹麦技术大学);School of Design(设计学院);University of Strathclyde(斯特拉斯克莱德大学);Department of Design, Manufacturing and Engineering Management(设计、制造与工程管理系)
API Secrets Should Never Become Tokens in the LLM's Vocabulary: A Threat Analysis of API Credential Handling in LLM Agent Systems and an Empirical Evaluation of a Vault-Mediated Execution Boundary
Breaking the Protocol: Security Analysis of the Model Context Protocol Specification and Prompt Injection Vulnerabilities in Tool-Integrated LLM Agents
PersonaArena: Dynamic Simulation for Evaluating and Enhancing Persona-Level Role-Playing in Large Language Models
College of Computer Science and Software Engineering, Shenzhen University(深圳大学计算机科学与软件工程学院);Microsoft Research Asia(微软亚洲研究院);Microsoft Gaming(微软游戏);Provincial Key Laboratory of Intelligent Communication and Digital Society Governance, Shenzhen University(深圳大学省级智能通信与数字社会治理重点实验室)
Leveraging Metamemory Agent for Enhanced Data-Free Code Generation in Large Language Models
College of Computing and Data Science, Nanyang Technological University, Singapore(南洋理工大学计算机与数据科学学院,新加坡);School of Computer Science, National Engineering Research Center for Multimedia Software, Wuhan University, Wuhan, China(多媒体软件国家工程研究中心计算机学院,武汉大学,中国);The University of Sydney, Sydney, Australia(悉尼大学,澳大利亚);Yunnan United Vision Technology Company Ltd., China(云南联合视界技术有限公司,中国);School of Computer Science, Wuhan University, Wuhan, China(武汉大学计算机学院,中国);School of Information Science and Engineering, Yunnan University, Kunming, China(云南大学信息科学与工程学院,中国)
Which Type of Students can LLMs Act? Investigating Authentic Simulation with Graph-based Human-AI Collaborative System
College of AI, Tsinghua University(人工智能学院,清华大学);School of Education, Tsinghua University(教育学院,清华大学);Department of Statistics and Data Science, Tsinghua University(统计与数据科学系,清华大学);Department of Computer Science and Technology, Tsinghua University(计算机科学与技术系,清华大学)
CirrusBench: Evaluating LLM-based Agents Beyond Correctness in Real-World Cloud Service Environments
School of Mathematics and Sciences, Fudan University(复旦大学数学科学学院);Shanghai Center for Mathematical Sciences, Fudan University(复旦大学上海数学中心);Alibaba Group(阿里巴巴集团);Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University(复旦大学类脑智能科学与技术研究院);Center for Applied Mathematics & Shanghai Key Laboratory of Contemporary Applied Mathematics, Fudan University(复旦大学应用数学中心暨上海市现代应用数学重点实验室)