arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.02589cs.CL
收藏

利用多模态大型语言模型进行结合语境的漫画机器翻译

Context-Informed Machine Translation of Manga using Multimodal Large Language Models

Delft University of Technology(代尔夫特理工大学);Mantra Inc.(曼特拉公司)

Philip Lippmann, Konrad Skublicki, Joshua Tanner, Shonosuke Ishiwatari, Jie Yang

AI总结:

本文提出利用多模态 LLM 视觉上下文的漫画翻译方法,评估翻译单元与上下文长度,发布日波平行数据集和开源基准工具,并在日英、日波翻译上取得最优结果。

摘要:

由于手工翻译需要大量时间和精力,大多数漫画从未走出日本国内市场。自动漫画翻译是一种颇具前景的潜在解决方案。然而,这是一个新兴且发展不足的领域,其复杂性甚至高于标准翻译,因为需要在翻译过程中有效融入视觉元素以消除歧义。在本研究中,我们考察多模态大型语言模型(LLM)能在多大程度上提供有效的漫画翻译,从而帮助漫画作者和出版商触达更广泛的受众。具体而言,我们提出一种方法,利用多模态 LLM 的视觉组件来提高翻译质量,并评估翻译单元大小和上下文长度的影响,同时提出一种令牌高效的漫画翻译方法。此外,我们引入一个新的评估数据集——首个日波平行漫画翻译数据集——作为供未来研究使用的基准的一部分。最后,我们贡献了一套开源软件套件,使其他研究者能够对用于漫画翻译的 LLM 进行基准测试。研究结果表明,我们提出的方法在日英翻译上取得了最先进的结果,并为日波翻译设立了新标准。

英文摘要:

Due to the significant time and effort required for handcrafting translations, most manga never leave the domestic Japanese market. Automatic manga translation is a promising potential solution. However, it is a budding and underdeveloped field and presents complexities even greater than those found in standard translation due to the need to effectively incorporate visual elements into the translation process to resolve ambiguities. In this work, we investigate to what extent multimodal large language models (LLMs) can provide effective manga translation, thereby assisting manga authors and publishers in reaching wider audiences. Specifically, we propose a methodology that leverages the vision component of multimodal LLMs to improve translation quality and evaluate the impact of translation unit size, context length, and propose a token efficient approach for manga translation. Moreover, we introduce a new evaluation dataset -- the first parallel Japanese-Polish manga translation dataset -- as part of a benchmark to be used in future research. Finally, we contribute an open-source software suite, enabling others to benchmark LLMs for manga translation. Our findings demonstrate that our proposed methods achieve state-of-the-art results for Japanese-English translation and set a new standard for Japanese-Polish.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑