规范文档上的版本与范围感知问答：一个已部署系统及其在生产规模下的端到端评估
Version- and Scope-Aware Question Answering over Normative Documents: A Deployed System and an End-to-End Evaluation at Production Scale
浏览论文内容
中文总结 AI 辅助
针对规范文档问答，提出显式规则解决版本与范围的受管系统，在生产规模评估中得分97.7，显著优于托管服务（88.1），并已商业部署。
中文摘要 AI 辅助
正确回答基于规范文档的问题，往往依赖于单个段落之外的信息：检索到的文档是否为当前有效版本；它是否适用于所涉司法管辖区、主体（如机构或申请人）及日期；以及每条规范性主张能否追溯到其支持的源文本。托管检索服务已大幅降低了在此类语料库上构建初始系统的工程成本，使得“上传文档并提问”成为常见默认做法。我们在生产部署中提供的约73,000份候选规范文档上评估了这一默认做法。评估使用了我们已发布基准中200个问题的分层样本，每个问题都有黄金源文档；发布的抽样规则不读取任何系统输出或分数。我们将托管服务与一个在生成前通过显式规则解决版本和范围问题的受管系统进行比较。受管系统总体得分为97.7，而托管服务得分为88.1，差距为9.6分（基于未取整均值计算）。问题集、两个系统所评估的答案文本、分数以及用于复现所报告基准统计数据的脚本均已公开。受管配置自2026年1月起作为商业产品运营，服务1,126名注册用户；具名客户组织包括智谱AI和乐成健康。截至2026年4月中旬，其已达到每个工作日约10万次调用。
英文摘要
Correctly answering a question grounded in normative documents often depends on information outside any single passage: whether the retrieved document is the version currently in force; whether it applies to the jurisdiction, subject (such as an institution or applicant), and date at issue; and whether each normative claim can be traced to its supporting source text. Hosted retrieval services have substantially lowered the engineering cost of building an initial system over such corpora, making "upload the documents and ask" a common default. We evaluate this default on approximately 73,000 candidate normative documents supplied to a production deployment. The evaluation uses a stratified sample of 200 questions from our published benchmark, with a gold source document for every question; the released sampling rule reads no system outputs or scores. We compare the hosted service with a governed system that resolves version and scope through explicit rules before generation. The governed system scored 97.7 overall, while the hosted service scored 88.1, a gap of 9.6 points computed from unrounded means. The question set, the answer text evaluated for both systems, the scores, and the scripts used to reproduce the reported benchmark statistics are public. The governed configuration has operated as a commercial product since January 2026 and serves 1,126 registered users; named customer organizations include Zhipu AI and Lecheng Health. By mid-April 2026, it had reached roughly 100,000 calls per workday.
发表机构
- Beijing Caizhi Technology Co., Ltd.（北京才智科技有限公司）
- dknownAI
机构由 AI 辅助整理，请以论文原文为准。
