PCFBench：产品碳足迹估算的诊断基准

中文摘要 AI 辅助

AI系统正被部署到高风险、领域特定的工作流程中，这类流程不仅要求最终输出正确，还要求每一个中间步骤都正确。产品碳足迹（PCF，即归因于实物产品的温室气体排放量）估算就是这类工作流程之一。AI智能体越来越多地被用于生成PCF，但现有评估要么仅对总排放量评分（掩盖错误来源并抵消错误），要么仅对孤立的子任务评分（忽略组合交互）。我们推出PCFBench，这是首个将PCF建模拆分为可独立评估任务的基准，这些任务需要分解、检索、本体匹配和数值提取。它包含6项任务共614个专家标注条目，共同探究了欠规范、冲突上下文和数值约束下的推理。对来自4家提供商的8种前沿大语言模型（LLM）而言，没有任何单一模型占据主导。尽管最强模型能在77%的产品上将总产品排放量估算至申报总量的2倍以内，但当PCF逐步生成时，这一比例降至37%-58%，且仅有45%-75%的模型符合质量守恒定律。这些缺陷破坏了从业者比较产品和推动脱碳所需的透明度。我们发布该数据集和评估工具包以支持针对性的研究进展。

英文摘要

AI systems are being deployed on high-stakes, domain-specific workflows that demand correctness not just in the final output, but at every intermediate step. One such workflow is estimating a product carbon footprint (PCF), the greenhouse-gas emissions attributable to a physical product. AI agents are increasingly being used to generate PCFs, but existing evaluations score either total emissions (hiding error sources and cancelling mistakes) or sub-tasks in isolation (missing compositional interactions). We introduce PCFBench, the first benchmark to carve PCF modeling into independently-evaluable tasks that require decomposition, retrieval, ontology matching, and numerical extraction. It comprises 614 expert-labelled items across six tasks. Together they probe reasoning under under-specification, conflicting context, and numerical constraints. Across eight frontier LLMs from four providers, no single model dominates. Although the strongest models estimate total product emissions within 2 times of declared totals on 77% of products, this rate drops to 37-58% when the PCF is generated step by step, with only 45-75% obeying mass conservation. These failures undermine the transparency practitioners need to compare products and drive decarbonization. We release the dataset and evaluation harness to support targeted progress.

发表机构

• Watershed Technology, Inc.（沃特斯hed科技公司）

机构由 AI 辅助整理，请以论文原文为准。