ComfyBench：在ComfyUI中基于LLM的代理自主设计协作AI系统的基准测试
ComfyBench: Benchmarking LLM-based Agents in ComfyUI for Autonomously Designing Collaborative AI Systems
AI总结：
ComfyBench通过评估基于LLM的代理在ComfyUI中自主设计协作AI系统的能力，提出ComfyAgent框架，展示其在任务解决中的性能与潜力。
摘要：
以往的AI研究大多集中在开发单体模型以最大化其智能，主要目标是提高在特定任务上的性能。相比之下，本研究试图研究使用基于LLM的代理来自主设计协作AI系统。为探讨这一问题，我们首先介绍了ComfyBench，用于评估代理在ComfyUI中设计协作AI系统的能力。ComfyBench是一个包含200种多样任务的综合性基准，涵盖各种遵循指令生成的挑战，同时包含对3,205个节点和20个流程的详细注释。基于ComfyBench，我们进一步开发了ComfyAgent，一种新的框架，使基于LLM的代理能够通过生成流程来自主设计协作AI系统。ComfyAgent基于两个核心概念。首先，它用代码表示流程，可以被解释器逆向转换为流程并作为协作系统执行。其次，它构建了一个多代理系统，这些代理合作从现有流程中学习并为给定任务生成新流程。尽管实验结果表明，ComfyAgent在ComfyBench上实现了与o1-preview相当的解决率，并显著超越了其他代理，但ComfyAgent仅解决了15%的创造性任务。基于LLM的代理在自主设计协作AI系统方面仍有很长的路要走。ComfyBench的进展正在为更智能和自主的协作AI系统铺平道路。
英文摘要：
Much previous AI research has focused on developing monolithic models to maximize their intelligence, with the primary goal of enhancing performance on specific tasks. In contrast, this work attempts to study using LLM-based agents to design collaborative AI systems autonomously. To explore this problem, we first introduce ComfyBench to evaluate agents's ability to design collaborative AI systems in ComfyUI. ComfyBench is a comprehensive benchmark comprising 200 diverse tasks covering various instruction-following generation challenges, along with detailed annotations for 3,205 nodes and 20 workflows. Based on ComfyBench, we further develop ComfyAgent, a novel framework that empowers LLM-based agents to autonomously design collaborative AI systems by generating workflows. ComfyAgent is based on two core concepts. First, it represents workflows with code, which can be reversibly converted into workflows and executed as collaborative systems by the interpreter. Second, it constructs a multi-agent system that cooperates to learn from existing workflows and generate new workflows for a given task. While experimental results demonstrate that ComfyAgent achieves a comparable resolve rate to o1-preview and significantly surpasses other agents on ComfyBench, ComfyAgent has resolved only 15\% of creative tasks. LLM-based agents still have a long way to go in autonomously designing collaborative AI systems. Progress with ComfyBench is paving the way for more intelligent and autonomous collaborative AI systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
