当知识库成为黄金标准：测量实体级机器翻译中的资源共享评估循环
When the Knowledge Base Becomes the Gold Standard: Measuring Resource-Shared Evaluation Loops in Entity-Level Machine Translation
浏览论文内容
中文总结 AI 辅助
本研究针对实体级机器翻译中知识库作为黄金标准形成的资源共享评估循环，利用韩国国立历史研究院的专家注释测量该循环，发现KB增益仅存在于共享资源部分，指标反映模型属性。
中文摘要 AI 辅助
《承政院日记》是联合国教科文组织世界记忆名录中的文献，仅37.4%已被翻译，自动翻译中最明显的失败模式是人名——误读人名会破坏历史事实，而非仅停留在表层。低资源历史领域没有用于实体翻译的专家黄金标准，因此从业者用知识库（KB）替代黄金标准；该知识库正是注入系统的同一资源：评分变得自我参照，指标衡量的是指令合规性而非翻译质量。我们对这一循环进行测量：使用韩国国立历史研究院的专家人名注释作为独立于注入流程的黄金标准，固定实体集，仅变更正确文本的来源。在527个专家注释提及中，仅31.1%位于注入流程之外，剩余的循环并非均匀分布——在重叠部分，注入文本与人工翻译的一致性达97.8%，而在独立部分这一比例为70.1%，因此看似最健康的部分正是该循环所支撑的部分。对四个模型的双重差分分析显示，KB注入带来的增益仅局限于黄金标准与注入资源共享的部分；在独立部分，增益为零或更低。尽管基线能力相差五倍，注入后的保留率仍集中在0.910至0.996的狭窄区间内，因此报告的增益是先前性能的补集，较弱模型的提升看似更为显著。在通过移除构建过滤器构建的独立样本上，该指标在模型内（区间重叠）可重复，同时能区分不同模型（区间不重叠）——它反映的是模型的属性，而非样本的属性。
英文摘要
The Seungjeongwon Ilgi, a UNESCO Memory of the World record, is only 37.4% translated, and the most conspicuous failure mode in automatic translation is the person name -- a misread name corrupts the historical fact rather than merely the surface. Low-resource historical domains have no expert gold standard for entity translation, so practitioners substitute a knowledge base (KB) for the gold. That KB is the same resource injected into the system: scoring becomes self-referential and the metric measures instruction compliance rather than translation quality. We measure this loop. Using expert person-name annotations from the National Institute of Korean History as a gold independent of the injection pipeline, we hold the entity set fixed and vary only the provenance of the correct reading. Of 527 expert-annotated mentions, only 31.1% lie outside the injection pipeline, and the residual loop is not uniform -- in the overlapping segment the injected reading agrees with the human translation 97.8% of the time against 70.1% in the independent one, so the segment that looks healthiest is the one the loop is holding up. Across four models, a difference-in-differences analysis shows the gain from KB injection is confined to the segment whose gold shares the injected resource; in the independent segment it is at or below zero. Post-injection preservation clusters in a narrow 0.910-0.996 band even though baseline capability differs fivefold, so the reported gain is the complement of prior performance and weaker models appear to improve more dramatically. On an independent sample built by removing the construction filter, the measure replicates within model (overlapping intervals) while discriminating between models (non-overlapping intervals) -- it reflects a property of the model, not of the sample.
