JADE：面向开放式专业任务的专家基础动态评估
JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks
AI总结：
提出JADE双层评估框架，结合专家知识与动态声明级评估，解决开放式专业任务中严格性与灵活性的矛盾，在BizBench等基准上提升稳定性并揭示关键失败模式。
摘要：
在开放式专业任务上评估智能体AI面临着严格性与灵活性之间的根本困境。静态评分标准提供了严格、可重复的评估，但无法适应多样化的有效响应策略，而LLM作为评判者的方法虽能适应个体响应，却存在不稳定性和偏差。人类专家通过将领域基础原则与动态的声明级评估相结合来解决这一困境。受此过程启发，我们提出了\textbf{JADE}，一个双层评估框架。第一层将专家知识编码为预定义的评估技能集，提供稳定的评估标准。第二层执行报告特定的声明级评估，灵活评估多样化的推理策略，并通过证据依赖门控来使基于被反驳声明的结论无效。在BizBench上的实验表明，JADE提高了评估稳定性，并揭示了基于整体LLM的评估者遗漏的关键智能体失败模式。我们进一步展示了与专家编写的评分标准的高度一致性，并有效迁移到HealthBench和此http URL，涵盖医学和10个领域的专业评估设置。代码和数据可在以下网址获取：此https URL。
英文摘要：
Evaluating agentic AI on open-ended professional tasks faces a fundamental dilemma between rigor and flexibility. Static rubrics provide rigorous, reproducible assessment but fail to accommodate diverse valid response strategies, while LLM-as-a-judge approaches adapt to individual responses yet suffer from instability and bias. Human experts address this dilemma by combining domain-grounded principles with dynamic, claim-level assessment. Inspired by this process, we propose JADE, a two-layer evaluation framework. Layer 1 encodes expert knowledge as a predefined set of evaluation skills, providing stable evaluation criteria. Layer 2 performs report-specific, claim-level evaluation to flexibly assess diverse reasoning strategies, with evidence-dependency gating to invalidate conclusions built on refuted claims. Experiments on BizBench show that JADE improves evaluation stability and reveals critical agent failure modes missed by holistic LLM-based evaluators. We further demonstrate strong alignment with expert-authored rubrics and effective transfer to HealthBench and DR.BENCH, covering medical and 10-domain professional evaluation settings. Code and data are available at https://github.com/smiling-world/JADE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
