记录是任务的一部分：跨维护、安全与召回报告中的文本分类器匹配记录评估
The record is part of the task: matched-record evaluation of text classifiers across maintenance, safety and recall reporting
浏览论文内容
中文总结 AI 辅助
本研究将记录选择视为评估任务的一部分，在GE维修、NASA ASRS和NHTSA召回系统中比较匹配记录，发现记录差异对分类性能影响显著，建议在决策点信息上评估并报告记录和标签来源。
中文摘要 AI 辅助
许多操作案例在不同工作流程阶段和不同目的下被多次记录，但模型评估通常在模型比较开始前选择这些记录之一。我们将该选择视为评估的一部分，并在固定标签和划分下比较同一案例的匹配记录，涉及三个系统：GE航空航天维修事件、NASA ASRS安全报告和NHTSA车辆召回。在GE的三个字段中，对于标签独立于叙述而来源于零件交易的事件，留出宏F1分数范围从0.33到0.91。客户报告（在车间工作前撰写）与技术员报告（在诊断后但在生成标签的交易前撰写）之间的差异为0.46。该差异显著大于在同一事件上测试的表示和架构差异。公共系统显示出不同模式：NHTSA缺陷摘要保持最强，在每种测试的模型族下均如此，而ASRS分析师摘要在学习序列模型下优于记者叙述，但在词汇基线模型下则不然。二次分析表明，一些模型比较也依赖于记录。评估应在预期决策点可用的信息上运行，并应报告记录和标签是如何产生的。
英文摘要
Many operational cases are documented more than once, at different workflow stages and for different purposes, yet model evaluations normally select one of these records before model comparison begins. We treat that selection as part of the evaluation and compare matched records of the same cases under fixed labels and splits in three systems: GE Aerospace repair events, NASA ASRS safety reports and NHTSA vehicle recalls. Across the three GE fields, for events whose label comes from parts transactions independently of the narratives, held-out macro-F1 ranged from 0.33 to 0.91. A difference of 0.46 separated the customer report, written before shop work, from the technician report, written after diagnosis but before the transaction that generates the label. That difference is substantially larger than the representation and architecture differences tested on the same events. The public systems showed different patterns: the NHTSA defect summary remained strongest under every model family tested, whereas the ASRS analyst synopsis outperformed the reporter narrative under learned sequence models but not under lexical baselines. Secondary analyses showed that some model comparisons were also record-dependent. Evaluations should be run on the information available at the intended decision point and should report how both the record and the label were produced.
发表机构
- University of the West of England（西英格兰大学）
- Birzeit University（比尔宰特大学）
- GE Aerospace（通用电气航空航天）
机构由 AI 辅助整理，请以论文原文为准。
