基于$k$近邻与自动评估的特定模型质量估计
Quality Estimation with $k$-nearest Neighbors and Automatic Evaluation for Model-specific Quality Estimation
AI总结:
提出基于$k$近邻的无监督特定模型质量估计方法$k$NN-QE,并设计了使用基于参考指标作为黄金标准的自动评估方法验证其有效性。
摘要:
在机器翻译(MT)输出时提供质量分数,即无参考的质量估计(QE),对于告知用户翻译的可靠性至关重要。我们提出了一种特定模型的无监督QE方法,称为$k$NN-QE,它利用$k$近邻从MT模型的训练数据中提取信息。评估特定模型QE的性能并非易事,因为它们在自身的MT输出上提供质量分数,因此无法使用包含预制MT输出上人类质量分数的基准QE测试集进行评估。因此,我们提出了一种自动评估方法,该方法使用基于参考的指标的质量分数作为黄金标准,而不是人类生成的分数。我们首次进行了详细分析并得出结论,这种自动方法是足够的,且基于参考的MetricX-23最适合该任务。
英文摘要:
Providing quality scores along with Machine Translation (MT) output, so-called reference-free Quality Estimation (QE), is crucial to inform users about the reliability of the translation. We propose a model-specific, unsupervised QE approach, termed $k$NN-QE, that extracts information from the MT model's training data using $k$-nearest neighbors. Measuring the performance of model-specific QE is not straightforward, since they provide quality scores on their own MT output, thus cannot be evaluated using benchmark QE test sets containing human quality scores on premade MT output. Therefore, we propose an automatic evaluation method that uses quality scores from reference-based metrics as gold standard instead of human-generated ones. We are the first to conduct detailed analyses and conclude that this automatic method is sufficient, and the reference-based MetricX-23 is best for the task.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。