论动物通信翻译器的非交互式评估
On Non-interactive Evaluation of Animal Communication Translators
AI总结:
本研究提出一种无需互动或观测、仅基于英语输出的非交互式评估方法,利用逐段翻译和洗牌测试识别幻觉,以评估动物通信翻译器,并在数据稀缺实验中验证其有效性。
摘要:
如果你有一个AI鲸鱼到英语的翻译器,你如何验证它是否正常工作?是否需要与动物互动,或依赖诸如温度之类的接地观测?我们提供了理论和概念验证实验证据,表明对于足够复杂的语言,互动甚至观测可能并非必要。人们或许能够仅通过翻译器的英语输出对其进行评估,从而在安全性、伦理和成本方面提供潜在优势。这是在没有可用参考翻译的情况下进行机器翻译质量评估(MTQE)的一个实例。一个关键挑战是识别“幻觉”,即可能看起来流畅且合理的虚假翻译。我们提出使用逐段翻译结合经典的自然语言处理洗牌测试来评估翻译器。其思路是逐轮翻译动物通信,并评估所得翻译在按顺序排列时比打乱顺序更有意义的频率。在数据稀缺的人类语言和构造语言上进行的概念验证实验证明了这种评估方法的潜在效用。这些人类语言实验仅用于在数据稀缺情况下验证我们的无参考指标。结果发现,该指标与基于参考翻译的标准评估高度相关,而参考翻译在我们的实验中是可用的。我们还进行了理论分析,表明在翻译学习的早期阶段,互动可能既非必要也非高效。
英文摘要:
If you had an AI Whale-to-English translator, how could you validate whether or not it is working? Does one need to interact with the animals or rely on grounded observations such as temperature? We provide theoretical and proof-of-concept experimental evidence suggesting that interaction and even observations may not be necessary for sufficiently complex languages. One may be able to evaluate translators solely by their English outputs, offering potential advantages in terms of safety, ethics, and cost. This is an instance of machine translation quality evaluation (MTQE) without any reference translations available. A key challenge is identifying ``hallucinations,'' false translations which may appear fluent and plausible. We propose using segment-by-segment translation together with the classic NLP shuffle test to evaluate translators. The idea is to translate animal communication, turn by turn, and evaluate how often the resulting translations make more sense in order than permuted. Proof-of-concept experiments on data-scarce human languages and constructed languages demonstrate the potential utility of this evaluation methodology. These human-language experiments serve solely to validate our reference-free metric under data scarcity. It is found to correlate highly with a standard evaluation based on reference translations, which are available in our experiments. We also perform a theoretical analysis suggesting that interaction may not be necessary nor efficient in the early stages of learning to translate.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。