Orla:一个用于服务基于LLM的多智能体系统的库
Orla: A Library for Serving LLM-Based Multi-Agent Systems
AI总结:
Orla提供了一种通用抽象,将请求执行与工作流级策略分离,通过阶段映射器、工作流编排器和内存管理器实现智能体级控制,展示了在客户支持工作流中的应用,并通过两个数据集验证了其在延迟和成本上的改进。
摘要:
Orla引入了用于构建和运行基于LLM的代理系统库。现代代理应用由结合多个LLM推理步骤、工具调用和异构基础设施的工作流组成。目前,开发者通常通过手动组合 orchestration 代码与LLM服务引擎和工具执行逻辑来构建这些系统。Orla提供了一种通用抽象,将请求执行与工作流级策略分离。它作为现有LLM推理引擎上的服务层:开发者定义由阶段组成的流程,而Orla管理这些阶段如何映射、执行和跨模型和后端协调。通过三个机制提供代理级控制:阶段映射器,将每个阶段分配给合适的模型和后端;工作流编排器,调度阶段并管理其资源和上下文;以及内存管理器,管理跨工作流边界的推理状态,如KV缓存。我们通过一个客户支持工作流演示了Orla,展示了其许多功能。我们对两个数据集进行了评估,显示阶段映射相比单模型vLLM基线在延迟和成本上有所改进,而工作流级缓存管理减少了首次令牌生成时间。
英文摘要:
We introduce Orla, a library for constructing and running LLM-based agentic systems. Modern agentic applications consist of workflows that combine multiple LLM inference steps, tool calls, and heterogeneous infrastructure. Today, developers typically build these systems by manually composing orchestration code with LLM serving engines and tool execution logic. Orla provides a general abstraction that separates request execution from workflow-level policy. It acts as a serving layer above existing LLM inference engines: developers define workflows composed of stages, while Orla manages how those stages are mapped, executed, and coordinated across models and backends. It provides agent-level control through three mechanisms: a stage mapper, which assigns each stage to an appropriate model and backend; a workflow orchestrator, which schedules stages and manages their resources and context; and a memory manager, which manages inference state such as the KV cache across workflow boundaries. We demonstrate Orla with a customer support workflow that exercises many of its capabilities. We evaluate Orla on two datasets, showing that stage mapping improves latency and cost compared to a single-model vLLM baseline, while workflow-level cache management reduces time-to-first-token.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。