IntellAgent：用于评估对话式AI系统的多智能体框架
IntellAgent: A Multi-Agent Framework for Evaluating Conversational AI Systems
AI总结：
IntellAgent是一个开源多智能体框架，通过图建模和模拟生成合成基准，提供细粒度诊断，全面评估对话式AI系统性能并识别优化方向。
摘要：
大型语言模型（LLMs）正在变革人工智能，演变为能够自主规划和执行的任务导向型系统。LLMs的主要应用之一便是对话式AI系统，这些系统必须处理多轮对话、集成领域特定的API，并遵守严格的政策约束。然而，评估这些智能体仍然是一个重大挑战，因为传统方法无法捕捉真实世界交互的复杂性和多变性。我们引入了IntellAgent，这是一个可扩展、开源的多智能体框架，旨在全面评估对话式AI系统。IntellAgent通过结合策略驱动的图建模、真实事件生成和交互式用户-智能体模拟，自动创建多样化的合成基准。这种创新方法提供了细粒度的诊断，解决了静态和人工策划基准在粗粒度指标上的局限性。IntellAgent代表了评估对话式AI的范式转变。通过在不同复杂度级别上模拟真实的多策略场景，IntellAgent捕捉了智能体能力和策略约束之间微妙的相互作用。与传统方法不同，它采用基于图的策略模型来表示策略交互的关系、可能性和复杂性，从而实现高度详细的诊断。IntellAgent还识别关键的性能差距，为有针对性的优化提供可操作的见解。其模块化、开源的设计支持无缝集成新领域、策略和API，促进了可复现性和社区协作。我们的研究结果表明，IntellAgent通过解决连接研究与部署的挑战，成为推进对话式AI的有效框架。该框架可在https://github.com/plurai-ai/intellagent获取。
英文摘要：
Large Language Models (LLMs) are transforming artificial intelligence, evolving into task-oriented systems capable of autonomous planning and execution. One of the primary applications of LLMs is conversational AI systems, which must navigate multi-turn dialogues, integrate domain-specific APIs, and adhere to strict policy constraints. However, evaluating these agents remains a significant challenge, as traditional methods fail to capture the complexity and variability of real-world interactions. We introduce IntellAgent, a scalable, open-source multi-agent framework designed to evaluate conversational AI systems comprehensively. IntellAgent automates the creation of diverse, synthetic benchmarks by combining policy-driven graph modeling, realistic event generation, and interactive user-agent simulations. This innovative approach provides fine-grained diagnostics, addressing the limitations of static and manually curated benchmarks with coarse-grained metrics. IntellAgent represents a paradigm shift in evaluating conversational AI. By simulating realistic, multi-policy scenarios across varying levels of complexity, IntellAgent captures the nuanced interplay of agent capabilities and policy constraints. Unlike traditional methods, it employs a graph-based policy model to represent relationships, likelihoods, and complexities of policy interactions, enabling highly detailed diagnostics. IntellAgent also identifies critical performance gaps, offering actionable insights for targeted optimization. Its modular, open-source design supports seamless integration of new domains, policies, and APIs, fostering reproducibility and community collaboration. Our findings demonstrate that IntellAgent serves as an effective framework for advancing conversational AI by addressing challenges in bridging research and deployment. The framework is available at https://github.com/plurai-ai/intellagent
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
