Counsel: 面向智能体任务的元评估数据集
Counsel: A Meta-Evaluation Dataset for Agentic Tasks
AI总结：
提出Counsel数据集，通过人工标注对LLM评判的智能体轨迹错误进行元评估，用于校准和改进LLM评判器。
摘要：
随着智能体系统处理日益复杂的多步骤任务，评估其轨迹成为主要瓶颈——在流行的智能体基准上，人工标注单条轨迹可能需要数小时，使得为衡量性能或整理训练数据而扩展评估变得困难。这导致广泛依赖自动化方法，如LLM-as-a-judge (LLMJ) 在过程和结果层面大规模评判智能体，然而LLMJ评判的合理性往往未被衡量。在此，我们引入Counsel，首个面向智能体任务的元评估公共数据集。Counsel包含来自开放权重LLMJ在两个智能体基准（tau-bench（客服智能体）和DA-Code（编码智能体））上的过程级评判，以及这些评判的人工元评估。人工标注者将每个标记错误的评判标记为“准确”、“位置正确但推理差”或“不应标记”，实现了可靠的标注者间一致性（Krippendorff's alpha为0.78）。所得数据集根据人类对齐程度，在轨迹内错误位置和推理质量两个维度对LLMJ评判进行分层，可作为校准、改进或训练面向智能体的LLMJ的宝贵数据。比较开放权重评判器，我们发现更强的评判模型和更多的推理努力均能改善与人类的一致性，最强评判器在位置上达到约88%的一致性，在推理上达到约65%。Counsel使用开放权重模型生成，并采用宽松许可供社区广泛使用，我们希望这将促进对基于LLM的智能体系统评估器的严谨研究和改进对齐。
英文摘要：
As agentic systems tackle increasingly complex multi-step tasks, evaluating their trajectories presents a major bottleneck - human annotation of a single trajectory on popular agentic benchmarks can take hours, making it difficult to scale evaluations for measuring performance or curating training data. This has driven widespread reliance on automated approaches such as LLM-as-a-judge (LLMJ) to critique agents at the process and outcome-levels at scale, however, the soundness of LLMJ critiques often goes unmeasured. Here, we introduce Counsel, the first public dataset of meta-evaluations for agentic tasks. Counsel consists of process-level critiques from open-weight LLMJs on two agent benchmarks: tau-bench (customer support agents) and DA-Code (coding agents), and human meta-evaluations of these critiques. Human annotators label critiques on each flagged error as "spot on", "correct location but poor reasoning", or "should not have flagged", achieving reliable inter-annotator agreement (Krippendorff's alpha of 0.78). The resulting dataset stratifies LLMJ critiques by human alignment across both error location within a trajectory and reasoning quality, serving as valuable data to calibrate, improve, or train LLMJs for agents. Comparing open-weight judges, we find that more capable judge models and more reasoning effort both enabled improved human agreement, with the strongest judge reaching ~88% agreement on location and ~65% on reasoning. Counsel is generated using open-weight models and is permissively licensed for broad community use, which we hope will enable rigorous study and improved alignment of LLM-based evaluators for agentic systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
