端到端生成系统中字幕分割的评估
Evaluating Subtitle Segmentation for End-to-end Generation Systems
摘要：
字幕在屏幕上显示为简短的文本，根据形式约束（长度）和语法/语义标准进行分割。可以用序列分割度量针对人类参考来评估字幕分割。然而，当系统生成与参考不同的输出时，例如，对于端到端字幕系统，不能应用标准分割度量。在本文中，我们研究如何进行基于参考的评估分割精度，而不管文本内容。我们首先对评估字幕分割的现有指标进行系统分析。然后，我们引入$Sigma$，一个新的字幕分割分数来自一个近似的上限的BLEU分割边界，这使我们能够解开良好的分割效果的文本质量。为了比较$Sigma$与现有的度量，我们进一步提出了一个边界投影方法从不完美的假设到真正的参考。结果表明，所有的指标都能够奖励高质量的输出，但类似的输出系统的排名取决于每个指标的错误类型的敏感性。我们的全面分析表明，$Sigma$是一个有前途的分割候选人，但其可靠性超过其他分割指标仍有待验证，通过与人类的判断。
英文摘要：
Subtitles appear on screen as short pieces of text, segmented based on formal constraints (length) and syntactic/semantic criteria. Subtitle segmentation can be evaluated with sequence segmentation metrics against a human reference. However, standard segmentation metrics cannot be applied when systems generate outputs different than the reference, e.g. with end-to-end subtitling systems. In this paper, we study ways to conduct reference-based evaluations of segmentation accuracy irrespective of the textual content. We first conduct a systematic analysis of existing metrics for evaluating subtitle segmentation. We then introduce $Sigma$, a new Subtitle Segmentation Score derived from an approximate upper-bound of BLEU on segmentation boundaries, which allows us to disentangle the effect of good segmentation from text quality. To compare $Sigma$ with existing metrics, we further propose a boundary projection method from imperfect hypotheses to the true reference. Results show that all metrics are able to reward high quality output but for similar outputs system ranking depends on each metric's sensitivity to error type. Our thorough analyses suggest $Sigma$ is a promising segmentation candidate but its reliability over other segmentation metrics remains to be validated through correlations with human judgements.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
