SpeechAgents:基于多模态多智能体系统的人类交流模拟
SpeechAgents: Human-Communication Simulation with Multi-Modal Multi-Agent Systems
AI总结:
本文提出SpeechAgents,一个基于多模态LLM的多智能体系统,用于模拟人类交流,通过多模态信号作为消息媒介并引入多智能体微调增强能力,在自建基准上展现出优异的对话模拟效果与扩展性。
摘要:
人类交流是一个复杂且多样的过程,不仅涉及语言、常识和文化背景等多种因素,还需要语音等多模态信息的参与。基于大型语言模型(LLM)的多智能体系统在模拟人类社会方面展现出了良好的性能。我们能否利用基于LLM的多智能体系统来模拟人类交流?然而,当前的基于LLM的多智能体系统主要依赖文本作为主要媒介。在本文中,我们提出了SpeechAgents,这是一个专为模拟人类交流而设计的基于多模态LLM的多智能体系统。SpeechAgents利用多模态LLM作为单个智能体的控制中心,并采用多模态信号作为智能体间交换消息的媒介。此外,我们提出了多智能体微调(Multi-Agent Tuning),在不损害通用能力的前提下增强LLM的多智能体能力。为了加强和评估人类交流模拟的有效性,我们构建了人类交流模拟基准(Human-Communication Simulation Benchmark)。实验结果表明,SpeechAgents能够模拟内容一致、节奏真实且情感丰富的人类交流对话,即使在多达25个智能体的情况下也展现出出色的可扩展性,可应用于戏剧创作和有声小说生成等任务。代码和模型将在 https://github.com/0nutation/SpeechAgents 开源。
英文摘要:
Human communication is a complex and diverse process that not only involves multiple factors such as language, commonsense, and cultural backgrounds but also requires the participation of multimodal information, such as speech. Large Language Model (LLM)-based multi-agent systems have demonstrated promising performance in simulating human society. Can we leverage LLM-based multi-agent systems to simulate human communication? However, current LLM-based multi-agent systems mainly rely on text as the primary medium. In this paper, we propose SpeechAgents, a multi-modal LLM based multi-agent system designed for simulating human communication. SpeechAgents utilizes multi-modal LLM as the control center for individual agent and employes multi-modal signals as the medium for exchanged messages among agents. Additionally, we propose Multi-Agent Tuning to enhance the multi-agent capabilities of LLM without compromising general abilities. To strengthen and evaluate the effectiveness of human communication simulation, we build the Human-Communication Simulation Benchmark. Experimental results demonstrate that SpeechAgents can simulate human communication dialogues with consistent content, authentic rhythm, and rich emotions and demonstrate excellent scalability even with up to 25 agents, which can apply to tasks such as drama creation and audio novels generation. Code and models will be open-sourced at https://github. com/0nutation/SpeechAgents
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。