arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.21956cs.CV
收藏

全局-局部双感知用于高分辨率文本密集图像翻译的MLLMs

Global-Local Dual Perception for MLLMs in High-Resolution Text-Rich Image Translation

School of Computer Science and Technology, East China Normal University(东华大学计算机科学与技术学院);Labs, Huawei Technologies Co., LTD(华为技术有限公司2012实验室);School of Automation and Intelligent Sensing, Shanghai Jiao Tong University(上海交通大学自动化与智能感知学院)

Junxin Lu, Tengfei Song, Zhanglin Wu, Pengfei Li, Xiaowei Liang, Hui Yang, Kun Chen, Ning Xie, Yunfei Lu, Jing Zhao, Shiliang Sun, Daimeng Wei

AI总结:

本文提出GLoTran框架,通过全局-局部双感知方法提升高分辨率文本密集图像翻译的准确性和完整性。

摘要:

文本图像机器翻译(TIMT)旨在将图像中嵌入的文本从源语言翻译成目标语言,需要视觉感知和语言理解的协同整合。现有TIMT方法,无论是级联流水线还是端到端的多模态大语言模型(MLLMs),在处理高分辨率文本密集图像时面临挑战,由于布局杂乱、字体多样和非文本干扰,导致文本缺失、语义漂移和上下文不一致。为了解决这些挑战,我们提出了GLoTran,一种针对基于MLLMs的TIMT的全局-局部双视觉感知框架。GLoTran整合了低分辨率全局图像和多尺度区域级文本图像切片,在指令引导对齐策略下,使MLLMs在保持场景级上下文一致性的同时,忠实捕捉细粒度的文本细节。此外,为了实现这种双感知范式,我们构建了GLoD,一个大规模的文本密集TIMT数据集,包含510,000对高分辨率全局-局部图像-文本对,覆盖多样化的现实场景。广泛实验表明,GLoTran在翻译完整性和准确性上显著优于最先进的MLLMs,为高分辨率和文本密集条件下的细粒度TIMT提供了一种新范式。

英文摘要:

Text Image Machine Translation (TIMT) aims to translate text embedded in images in the source-language into target-language, requiring synergistic integration of visual perception and linguistic understanding. Existing TIMT methods, whether cascaded pipelines or end-to-end multimodal large language models (MLLMs),struggle with high-resolution text-rich images due to cluttered layouts, diverse fonts, and non-textual distractions, resulting in text omission, semantic drift, and contextual inconsistency. To address these challenges, we propose GLoTran, a global-local dual visual perception framework for MLLM-based TIMT. GLoTran integrates a low-resolution global image with multi-scale region-level text image slices under an instruction-guided alignment strategy, conditioning MLLMs to maintain scene-level contextual consistency while faithfully capturing fine-grained textual details. Moreover, to realize this dual-perception paradigm, we construct GLoD, a large-scale text-rich TIMT dataset comprising 510K high-resolution global-local image-text pairs covering diverse real-world scenarios. Extensive experiments demonstrate that GLoTran substantially improves translation completeness and accuracy over state-of-the-art MLLMs, offering a new paradigm for fine-grained TIMT under high-resolution and text-rich conditions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑