手语翻译自动评估的批判性研究
A Critical Study of Automatic Evaluation in Sign Language Translation
AI总结：
本研究针对手语翻译现有文本类自动评估指标可靠性不明的问题，对比分析传统词汇重叠指标与大语言模型类评估器在三类受控条件下的表现，指出两类指标的局限，提出需构建多模态评估框架。
摘要：
自动评估指标对于推动手语翻译（SLT）的发展至关重要。当前的SLT评估指标如BLEU、ROUGE等仅基于文本，这类基于文本的指标能在多大程度上可靠衡量SLT输出质量尚不明确。为填补这一空白，本研究分析了六类指标以探究文本类SLT评估指标的局限性，一方面涵盖BLEU、chrF、ROUGE以及BLEURT，另一方面包括G-Eval、GEMBA零样本直接评估等基于大语言模型（LLM）的评估器。具体而言，我们在释义改写、模型输出幻觉、句长变化三种受控条件下评估了这些指标的一致性与鲁棒性。分析结果凸显了词汇重叠类指标的局限性，同时证明：尽管基于LLM的评估器能更好捕捉传统指标常遗漏的语义等价性，但它们也会对经LLM释义的译文表现出偏好偏差。此外，尽管所有指标都能检测幻觉，BLEU往往过于敏感，而BLEURT与基于LLM的评估器对细微幻觉案例则相对宽松。这一发现表明，亟需拓展文本类指标之外的多模态评估框架，以实现对SLT输出更全面的评估。
英文摘要：
Automatic evaluation metrics are crucial for advancing sign language translation (SLT). Current SLT evaluation metrics, such as BLEU and ROUGE, are only text-based, and it remains unclear to what extent text-based metrics can reliably capture the quality of SLT outputs. To address this gap, we investigate the limitations of text-based SLT evaluation metrics by analyzing six metrics, including BLEU, chrF, and ROUGE, as well as BLEURT on the one hand, and large language model (LLM)-based evaluators such as G-Eval and GEMBA zero-shot direct assessment on the other hand. Specifically, we assess the consistency and robustness of these metrics under three controlled conditions: paraphrasing, hallucinations in model outputs, and variations in sentence length. Our analysis highlights the limitations of lexical overlap metrics and demonstrates that while LLM-based evaluators better capture semantic equivalence often missed by conventional metrics, they can also exhibit bias toward LLM-paraphrased translations. Moreover, although all metrics are able to detect hallucinations, BLEU tends to be overly sensitive, whereas BLEURT and LLM-based evaluators are comparatively lenient toward subtle cases. This motivates the need for multimodal evaluation frameworks that extend beyond text-based metrics to enable a more holistic assessment of SLT outputs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
