arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.19583cs.SE
收藏

迈向面向架构的LLM代理评估指标

Toward Architecture-Aware Evaluation Metrics for LLM Agents

Federal University of Campina Grande(坎皮纳格兰德联邦大学)

Débora Souza, Patrícia Machado

AI总结:

本文提出了一种面向架构的LLM代理评估方法,通过连接代理组件与可观察行为及评估指标,提升评估的针对性和透明度。

摘要:

基于LLM的代理正在成为软件工程任务的核心,但评估它们仍然碎片化且主要以模型为中心。现有研究忽视了架构组件,如规划器、内存和工具路由器,如何塑造代理行为,限制了诊断能力。我们提出了一种轻量级、面向架构的方法,将代理组件与其可观察行为以及能够评估它们的指标联系起来。我们的方法明确了应测量什么以及为什么,我们通过现实世界的代理示例说明了其应用,使对基于LLM的代理的评估更加有针对性、透明和可操作。

英文摘要:

LLM-based agents are becoming central to software engineering tasks, yet evaluating them remains fragmented and largely model-centric. Existing studies overlook how architectural components, such as planners, memory, and tool routers, shape agent behavior, limiting diagnostic power. We propose a lightweight, architecture-informed approach that links agent components to their observable behaviors and to the metrics capable of evaluating them. Our method clarifies what to measure and why, and we illustrate its application through real world agents, enabling more targeted, transparent, and actionable evaluation of LLM-based agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑