Gavel: 智能体结合检查表评估LLM长上下文法律摘要
Gavel: Agent Meets Checklist for Evaluating LLMs on Long-Context Legal Summarization
AI总结：
提出Gavel框架，通过参考评估和免参考智能体评估12个前沿LLM在多文档法律摘要中的表现，发现模型易遗漏关键信息而非幻觉，且性能随案件长度增加而下降。
摘要：
大型语言模型（LLM）现在支持高达100万token的上下文，但它们在复杂长上下文任务中的优缺点仍不明确。为此，我们专注于多文档法律案例摘要，其中单个案例通常跨越许多文档，超过10万token。我们使用Gavel系统评估了12个前沿LLM，该系统包括Gavel-Ref（基于参考的评估框架，包含检查表、残差事实和写作风格评估）和Gavel-Agent（免参考智能体，直接根据源文档评估事实覆盖度）。我们的结果表明，当前模型更倾向于遗漏关键信息而非产生幻觉。它们在简单的检查表项目（如提交日期）上表现良好，但在罕见且复杂的项目（如和解）上表现不佳。性能也随案例长度增加而下降。为了元评估Gavel，我们收集了160小时的人工标注。与端到端和逐块方法相比，Gavel-Agent在实现竞争性能的同时，减少了至少36%的token使用。Gavel-Agent还能泛化到医学领域，以至少减少77%的token实现最佳性能。
英文摘要：
Large language models (LLMs) now support contexts of up to 1M tokens, but their strengths and weaknesses on complex long-context tasks remain unclear. To study this, we focus on multi-document legal case summarization, where a single case often spans many documents exceeding 100K tokens. We systematically evaluate 12 frontier LLMs with Gavel, which consists of Gavel-Ref, a reference-based evaluation framework with checklist, residual-fact, and writing-style evaluations, and Gavel-Agent, a reference-free agent for evaluating factual coverage directly from source documents. Our results show that current models are more prone to omitting key information than hallucinating. They all perform well on simple checklist items, such as filing date, but struggle with rare and complex items, such as settlements. Performance also declines as case length increases. To meta-evaluate Gavel, we collect 160 hours of human annotations. Gavel-Agent reduces token usage by at least 36% compared to end-to-end and chunk-by-chunk methods while achieving competitive performance. Gavel-Agent also generalizes to the medical domain, performing the best with at least 77% fewer tokens.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
