arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.00456cs.AI
收藏

在保险承保环境中评估智能体

Benchmarking Agents in Insurance Underwriting Environments

Amanda Dsouza, Ramya Ramakrishnan, Charles Dickens, Bhavishya Pohani, Christopher M Glaze

AI总结:

本文提出UNDERWRITE基准,通过专家协作模拟真实保险承保场景,揭示了代理性能与企业需求间的差距,强调了专家参与和组合性方法的重要性。

摘要:

随着人工智能代理整合到企业应用中,其评估需要反映真实世界操作复杂性的基准。然而,现有基准过度强调开放领域,如代码,使用狭窄的准确性指标,并缺乏真实复杂性。我们提出了UNDERWRITE,一个以专家为导向、多轮次的保险承保基准,是在与领域专家紧密合作下设计的,以捕捉真实的企业挑战。UNDERWRITE引入了当前基准中常缺失的关键现实因素:专有业务知识、嘈杂的工具接口以及需要仔细信息收集的不完美的模拟用户。评估13个前沿模型,我们揭示了研究实验室性能与企业准备之间的显著差距:最准确的模型不一定是最高效的,模型在拥有工具访问权限的情况下仍会编造领域知识,且pass^k结果显示性能下降了20%。UNDERWRITE的结果表明,专家在基准设计中的参与对于真实的代理评估至关重要,常见的代理框架表现出易碎性,这会扭曲性能报告,而专门领域中的编造检测需要组合性方法。我们的工作为开发更符合企业部署需求的基准提供了见解。

英文摘要:

As AI agents integrate into enterprise applications, their evaluation demands benchmarks that reflect the complexity of real-world operations. Instead, existing benchmarks overemphasize open-domains such as code, use narrow accuracy metrics, and lack authentic complexity. We present UNDERWRITE, an expert-first, multi-turn insurance underwriting benchmark designed in close collaboration with domain experts to capture real-world enterprise challenges. UNDERWRITE introduces critical realism factors often absent in current benchmarks: proprietary business knowledge, noisy tool interfaces, and imperfect simulated users requiring careful information gathering. Evaluating 13 frontier models, we uncover significant gaps between research lab performance and enterprise readiness: the most accurate models are not the most efficient, models hallucinate domain knowledge despite tool access, and pass^k results show a 20% drop in performance. The results from UNDERWRITE demonstrate that expert involvement in benchmark design is essential for realistic agent evaluation, common agentic frameworks exhibit brittleness that skews performance reporting, and hallucination detection in specialized domains demands compositional approaches. Our work provides insights for developing benchmarks that better align with enterprise deployment requirements.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑